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1 .  EXECUTIVE  SUMMARY 


Background 

As  part  of  an  effort  to  improve  Development  Planning  in  Electronic  Systems 

Division,  Deputy  for  Development  Plans,  ALPHATECH  was  asked  to  study  the  Vanguard 

3 

process  and  to  focus  on  Vanguard  mission  and  program  analysis.  A  C  I  workshop 

held  in  early  FY85  as  part  of  this  project  concluded  that  the  standard  Vanguard 

3 

analytic  model  (AFSCP  80-3)  was  not  appropriate  for  C  missions  and  that  other 
models  and  techniques  should  be  investigated.  Subsequent  review  of  existing 
analytic  models  found  that  none  of  the  then  available  models  were  both  applicable 
and  practical  for  use  by  Vanguard  analysts. 

We  considered  several  approaches  for  constructing  new  models  that  could 
provide  an  applicable  framework  for  connecting  Measures  of  Effectiveness 
(MOEs)  in  a  mission  with  Measures  of  Performance  (MOPs)  for  C3  systems,  and  a 
practical  procedure  that  would  not  cost  as  much  to  implement  as  would  conven¬ 
tional  computer  modeling  techniques.  The  Subjective  Transfer  Function  (STF) 
modeling  method  was  chosen  as  the  most  promising  approach  for  further  testing. 

STF  is  a  quantitative  technique  developed  at  RAND  Corporation.  It  uses  a 
causal  tree  structure  to  trace  the  flow  of  information  through  the  levels  of  a 
command  and  control  system.  A  specific  measure  is  defined  for  each  factor  in 
the  tree.  Relationships  among  the  factors  at  each  branch  are  derived  through 
questionnaires  administered  to  groups  of  experts  who  have  operational  experi¬ 
ence  with  the  system  being  modeled.  The  relationship  at  each  node,  i.e.,  how 
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different  levels  of  each  input  factor  combine  to  provide  a  level  of  output,  is 
expressed  as  a  subjectively  derived,  numerical  function.  The  functions  for 
the  entire  tree  can  be  concatenated  into  an  overall  quantitative  expression 
which  connects  system  capabilities,  which  are  input  at  the  bottom  of  the  tree, 
with  mission  effectiveness,  output  at  the  top  of  the  tree. 

Results 

We  designed  an  experimental  test  of  STF  based  on  the  previous  successful 

RAND  experiences.  The  Strategic  C2  Vanguard  mission  area  was  chosen  for  the 

test,  and  ALPHATECH  analysts  made  several  visits  to  both  NORAD  and  SAC  Head¬ 
quarters  to  design  the  Strategic  C2  tree  and  subsequently  to  collect  quantita¬ 
tive  data.  During  the  course  of  the  test  we  adapted  the  STF  method  to  fit  the 

problem  and  spent  much  time  devising  tree  nodes  and  measures  and  redefining 
the  STF  method.  We  also  put  considerable  effort  into  development  of  automated 
tools  to  generate  questionnaires  and  reduce  data,  and  into  installing  the 
computational  results  in  a  user  friendly  tool  for  analysts.  Due  to  limited 
availability  of  Air  Force  personnel,  we  completed  only  part  of  the  test; 
nevertheless  we  were  able  to  identify  both  benefits  and  limitations  of  the 
STF  approach  to  modeling  C2  systems. 

This  was  an  initial  experiment;  much  more  was  learned  about  STF  as  we 
proceeded.  Much  of  our  early  design  and  tool  development  was  not  used  in 
our  scaled  down  test  of  STF.  More  experience  with  STF  and  its  application 
to  Vanguard,  more  intimate  knowledge  of  the  mission,  and  more  time  are  needed 
if  the  Air  Force  wishes  to  convert  our  concept  demonstration  of  STF  into  a 
Vanguard  support  tool  for  production  use.  We  do  not  think  that  STF  can  be 
used  by  Vanguard  analysts  without  technical  assistance  from  STF-experienced 
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consultants;  we  feel  that  the  STF  approach  increases  rather  than  obviates  the 
need  for  familiarity  with  the  mission,  and  we  found  that  STF  implementation 
cost  is  higher  than  we  had  hoped. 

Causal  Trees 

The  tree  we  developed,  with  its  associated  measures,  is  a  useful  model 
of  the  Strategic  C2  System.  We  found  that  respondents  at  the  operating  com¬ 
mands  were  quickly  able  to  understand  the  ideas  behind  the  STF  approach.  Our 
efforts  to  identify  and  quantify  C2  requirements  paralleled  similar  efforts 
by  NORAD/SpaceCmd  to  document  Strategic  mission  requirements. 

Data  Collection 

In  some  instances  the  novelty  of  using  STF  and  the  difficulties  of  ap¬ 
plying  the  method  to  an  evolving  C2  mission  left  us  with  less  useful  data  than 
we  had  anticipated.  Nonetheless,  we  were  successful  in  collecting  most  of  the 
data  for  NORAD  systems.  We  did  not  collect  data  about  SAC  systems.  An  addi¬ 
tional  visit  to  the  commands,  as  recommended  in  RAND’s  approach  to  STF,  would 
have  been  useful  to  refine  the  tree  structure  and  to  administer  sample  test 
questionnaires  prior  to  the  final  data  collection  effort. 

Functional  Relationships 

The  functions  derived  from  the  data  appear  to  be  reasonable  representa¬ 
tions  of  the  relationships  among  causal  factors  and  their  outputs.  However, 
we  feel  that  respondents  may  not  have  clearly  understood  all  the  new  concepts 
that  we  introduced  in  the  course  of  administering  the  questionnaires;  hence, 
the  relationships  may  be  incomplete  and  the  data  inaccurate. 
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The  Expert  Problem 


Normally  in  applying  STF,  respondents  to  the  questionnaires  are  chosen 
to  be  experts  operationally  experienced  in  their  portion  of  the  mission.  The 
experts  assume,  as  part  of  the  “context”  of  the  questionnaires,  reasonable 
values  for  factors  that  are  not  specified  and  manipulated  by  design  in  STF. 

We  found,  however,  that  there  was  no  obvious  context  for  the  still-evolving 
strategic  mission,  and  there  were  no  experts  for  such  contextual  factors  as 
future  requirements  or  standard  scenarios.  We  had  to  generate  much  of  the 
context  ourselves,  tabulating  requirements  and  scenarios  in  detail,  in  order 
to  make  data  collection  possible. 

The  Scope  of  Implementing  STF 

Implementation  of  STF  reorganizes  but  does  not  avoid  the  fundamental  jobs 
of  portraying  the  C2  System  as  a  coherent  whole,  understanding  and  writing 
down  requirements,  tying  them  together  into  an  integrated  plan  and  relating 
them  to  individual  systems  and  technologies  that  might  be  acquired. 


Conclusions 


1.  The  STF  causal  tree  model  is  a  good  framework  for  the  operating 
commands  and  ESD  to  use  when  talking  about  future  operational 
requirements  and  their  relationship  to  system  capabilities. 

For  this  reason  alone  it  is  worth  pursuing. 

2.  Checklists  are  a  useful  way  of  collecting  uniform  data  about 
acquisition  programs.  Checklist  data  can  also  easily  be  mani¬ 
pulated  with  automated  data  processing  tools.  Vanguard  analysts 
should  consider  checklists  as  part  of  the  Vanguard  data  call 
even  if  STF  is  not  implemented  further. 

3.  The  STF  method  of  collecting  quantitative  data  about  causal 
factors  from  experts  is  well  suited  for  static  factors  that  are 
well  known  to  operational  personnel.  However,  rapidly  evolving 
missions  with  undefined  future  requirements  that  use  advanced 
technologies  (e.g.,  SDI ,  cruise  missile  defense)  have  no 
experts  and  do  not  lend  themselves  to  easy  analysis  via  STF. 
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For  such  missions  a  group  or  team  comprised  of  current  opera¬ 
tions  personnel,  system  planning  personnel  and  technologists 
must  be  formed  to  provide  the  required  expertise.  In  addition, 
other  approaches  to  collecting  quantitative  data  about  relation 
ships  in  a  causal  model  may  be  better  suited  to  ESDfs  needs. 
Nevertheless,  even  rough  quantitative  data  collected  via  STF 
provides  useful  insights  into  the  potential  use  and  limits  of 
system  capabilities  and  the  sometimes  complementary  relation¬ 
ships  among  those  capabilities. 

4.  The  analytic  results  from  our  initial  test  of  STF  were  not 
rigorous  enough  to  provide  quantitative  support  for  decisions 
about  the  relative  value  of  acquisition  programs. 

5.  The  software  tool  developed  on  a  personal  computer  demonstrates 
that  simple  but  powerful  ideas  can  be  embedded  in  portable, 
easy-to-use  software  that  the  Vanguard  analyst  can  use  directly 

6.  Although  developing  an  STF  model  requires  commitment  of  con¬ 
siderable  resources  in  the  form  of  ESD,  MAJCOM  and  technical 
consultant  time,  it  still  involves  less  of  an  investment  than 
does  a  computer  based  analytic  model. 
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2 .  INTRODUCTION 


This  section  briefly  describes  ALPHATECHfs  project  for  ESD/XR  and  the 
activities  that  led  up  to  our  experimental  test  of  the  Subjective  Transfer 
Function  method  applied  to  Vanguard  analysis.  Task  4  of  that  project.  A 
more  complete  discussion  of  Tasks  1-3  is  provided  in  TR-226-2:  "C3I  Analysis 
Tools  for  Development  Planning,  Cumulative  Report,  Tasks  1-3"  (Jan  31,  1985). 
In  the  following,  an  exposition  of  the  STF  approach  to  modeling  C2  systems  is 
provided.  We  also  outline  our  parallel  effort  to  investigate  planning  deci¬ 
sion  aids  carried  out  by  installing  the  analytic  results  of  our  experiment  on 
a  personal  computer. 

BACKGROUND  OF  PROJECT 
Previous  Tasks 

In  Task  1,  we  provided  a  formal  representation  of  the  Vanguard 
process  using  the  IDEF  representation  language.  This  task  familiarized  us 
with  Vanguard,  providing  ESD/XR  with  a  representation  of  Vanguard  at  ESD 
which  was  useful  for  training  and  introducing  other  communities  to  Vanguard. 

Under  Task  2,  ALPHATECH  hosted  a  workshop  which  reviewed  the 
current  Vanguard  process  and  identified  shortcomings,  including  a  lack  of 
applicable  quantitative  models  and  problems  with  the  existing  analytic  model 
used  by  Vanguard  analysts.  One  particular  problem  is  that  the  requirements 
for  acquisition  programs  are  set  by  the  operational  community  and  expressed 
in  terms  of  operational  measures.  The  capabilities  developed 
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by  acquisition  programs  are  often  expressed  by  the  development  community  in 
terms  of  system  performance.  There  appear  to  be  no  relationships,  currently 
acceptable  to  both  communities,  that  map  system  performance  measures  into 
operational  measures.  Furthermore,  under  the  current  Vanguard  process,  the 
Vanguard  analyst  is  asked  to  express  all  deficiencies  and  all  program  contri¬ 
butions  in  terms  of  a  single  measure,  percent  of  task  accomplishment,  that 
does  not  allow  for  either  the  effects  of  task  interrelationships  or  the  non¬ 
linear  contributions  of  tasks  to  mission  effectiveness. 

It  would  be  useful  for  the  analyst  to  have  a  better  way  of  translating 
system  capability  inputs  into  mission  effectiveness  outputs,  i.e.,  a  transfer 
function.  Such  a  transfer  function  might  be  derived  from  a  computational 
model  that  incorporated  detailed  algorithms  capturing  the  relationships  among 
C3I  capabilities,  weapon  system  capabilities,  and  mission  effectiveness. 

In  Task  3,  we  surveyed  available  computational  models,  implemented  as 
computer  programs,  to  see  if  any  of  them  were  appropriate  for  the  Vanguard 
analysts’  needs.  It  was  found  that  most  models  focus  on  weapon  system  per¬ 
formance  and  do  not  incorporate  C3I  system  performance.  Those  that  do  incor¬ 
porate  C3I  are  impractical  in  that  they  require  too  many  resources  to  operate, 
or  are  inappropriate  because  they  do  not  cover  enough  of  the  C3I  systems  of 
interest.  In  the  long  term,  the  use  of  complex  computational  models  might  be 
a  worthwhile  objective  for  C3I  Vanguard  analysts;  but  in  the  near  term,  sim¬ 
pler  models  are  necessary. 

Goals 

There  were  several  major  goals  of  the  Task  4  plan.  First,  to 
develop  a  model  relating  system  capability  to  mission  effectiveness  which 
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was  practical  for  use  by  Vanguard  analysts  *  We,  and  our  ESD  sponsors,  wanted 
to  adapt  an  existing  approach;  but  as  we  found  in  Task  3,  existing  computer 
models  were  not  the  solution. 

Second,  to  test  the  implementation  of  the  model  selected  for 
one  Vanguard  C2  mission.  For  ESDfs  purposes,  a  practical  demonstration  is 
better  than  a  theoretical  investigation.  Part  of  this  goal  included 
installing  the  model  in  an  automated  decision  aid  for  analysts. 

Third,  to  evaluate  the  opportunity  for  using  the  model  in 
Vanguard  on  a  larger  scale.  As  a  consequence  of  this  last  goal,  we  imple¬ 
mented  STF  with  general  purpose  tools  and  tried  a  few  excursions  to  test 
alternative  ways  of  making  STF  work. 

Choice  of  STF 

STF:  Subjective  Transfer  Function  method  was  one  of  the  methodologies 

discussed  at  the  workshop,  and  received  a  very  favorable  hearing  (Rand 

publication  R-3021-AF,  July,  1984).  This  method  allows  the  analyst  to  struc¬ 
ture  a  mission  area  as  a  hierarchy  of  factors  and  outcomes,  and  to  investigate 

the  relationship  between  these  on  the  basis  of  (subjective)  expert  judgment. 
Because  of  the  use. of  expert  judgment  as  a  surrogate  for  more  formal  analytic 
relationships,  this  method  promised  to  provide  a  means  for  modeling  a  very 
broad  mission  area,  incorporating  many  diverse  factors,  with  substantially 
less  investment  in  time,  and  personnel  resources  than  is  required  by  other 
modeling  methods.  Because  of  the  hierarchical  nature  of  the  model,  this 
method  promised  to  relate  measures  of  performance,  at  lower  levels  of  the 
hierarchy,  to  measures  of  effectiveness  at  higher  levels.  Thus  it  seemed  an 
appealing  alternative  to  more  conventional  computer  modeling  techniques  such 
as  simulation. 
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THE  SUBJECTIVE  TRANSFER  FUNCTION  APPROACH 


Overview 

The  STF  method  was  developed  at  RAND  Corporation  to  capture  the  essen¬ 
tials  of  a  transfer  function,  relating  factors  to  outcomes,  by  using  the 
structured  judgments  of  experts  as  surrogates  for  a  computational  model.  STF 
allows  the  experts  consulted  to  express  mission  effectiveness  and  task  perform¬ 
ance  in  terras  of  the  quantitative  variables  that  they  deem  appropriate;  the 
method  is  not  restricted  to  a  percent  task  accomplishment  as  its  only  measure. 
STF  can  capture  both  the  nonlinearity  of  performance  contribution  and  the 
relationship  among  performance  parameters  on  several  tasks.  STF  could  also  be 
a  first  step  toward  the  subsequent  development  of  more  objective  computational 
models  if  such  models  are  deemed  appropriate. 

Causal  Trees 

A  test  of  the  STF  method  requires  the  construction  of  a  hierarchy  (tree) 
of  appropriate  quantitative  measures,  such  as  effectiveness,  performance,  and 
capability.  Quantitative  information  must  then  be  gathered  by  means  of  care¬ 
fully  designed  data  collection  procedures  for  each  level  of  the  tree;  opera¬ 
tional  personnel  should  be  consulted  whenever  possible.  From  this  data,  STF 
models  are  derived  for  each  level  of  the  tree  and  then  concatenated  into  a 
single  model,  the  subjective  transfer  function,  relating  capabilities  at  the 
bottom  to  mission  effectiveness  at  the  top.  This  model,  in  the  form  of  a  set 
of  interacting  algebraic  formulas,  becomes  the  analyst’s  tool  to  evaluate  pro¬ 
grams  and  identify  deficiencies  and  technology  opportunities.  For  example, 

Fig.  1  shows  a  single  node  of  the  tree  that  was  constructed  for  the  strategic 
defense  mission. 
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Qualitative  Time  for 

Messages  per  second  Levels  Modification 

Figure  1.  Node  for  Questionnaire  1111 

The  preceding  figure  and  the  related  measures  were  derived  in  discussion 
with  Missile  Warning  Center  (MWC)  personnel*  The  figure  indicates  that  the 
factors  of  interest  to  ESP  which  affect  the  completeness  of  MWC  information 
are  those  shown:  data  rate  (from  the  sensors  to  the  MWC  systems),  data  pro¬ 
cessing  capability,  display  capability  and  system  currency  (or  flexibility). 
Other  factors  also  affect  this  outcome,  but  these  other  factors  are  not  of 
interest  to  ESD  since  they  do  not  relate  to  C3I  systems.  During  data  collec¬ 
tion  we  asked  the  respondent  experts  to  keep  these  other  factors  in  mind  at  a 
reasonable  level,  but  to  focus  on  the  difference  that  different  levels  of  our 
particular  factors  make  to  the  outcome,  completeness  of  MWC  information. 

Measures 

Factors  that  are  used  to  describe  C^  systems  must  be  measurable.  For 
operator  experts  to  answer  questions  about  these  factors,  the  measures  have 
to  be  clearly  known  to  the  operators,  not  obscure  or  abstract  entities  that 
they  have  no  feeling  for.  Objectively  determined,  quantitative  measures  are 
preferred,  but  subjective  measures,  with  qualitative  levels,  may  be  used 
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where  appropriate.  In  Fig,  1,  display  capability  was  described  with  qualita¬ 
tive  levels. 

Data  Collection 

Having  determined  the  factors  of  interest,  the  appropriate  measures  and  a 
reasonable  range  of  levels  for  each  measure  (for  Messages  per  Second  we  chose 
,1,  1,  10  and  100),  the  next  step  is  to  design  and  administer  a  questionnaire 
eliciting  subjective  estimates  of  the  outcome  for  various  combinations  of 
input  factor  levels.  Appendix  D  shows  the  questionnaire  that  was  administered 
for  the  node  shown  in  Fig,  1, 

The  questionnaire  is  administered  to  a  group  of  experts,  in  this  case  to 
experienced  MWC  personnel.  The  questionnaire  session  begins  with  a  thorough 

discussion  of  the  outcome,  the  factors  and  the  measures ,  Foil  owing  this,  selected 
questions  are  discussed  by  the  group;  the  object  of  this  discussion  is  not  to 

decide  what  the  "right”  answer  is,  but  to  ensure  that  all  respondents  are 
operating  in  the  same  background  context.  Finally,  the  questionnaire  is  com¬ 
pleted  by  each  respondent  without  consulting  the  others. 

Quantitative  Relationships 

Analysis  of  the  completed  questionnaires,  using  statistical  techniques 
similar  to  curve  fitting,  allows  the  derivation  of  an  algebraic  relationship, 
the  transfer  function,  between  the  factors  and  the  outcome.  Often  this  rela¬ 
tionship  has  a  simple  additive,  multiplicative  or  averaging  form.  The  method 
does  not  claim  to  explain  how  different  factor  levels  "cause"  a  particular 
level  of  outcome.  What  the  method  does  claim  is  that,  for  any  combination 
of  factor  levels,  within  the  range  covered  by  the  questionnaire,  the  transfer 
function  gives  a  good  approximation  to  the  subjective  estimate  of  the  outcome 
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level  that  the  respondents  would  have  given,  if  they  had  been  asked  that  par¬ 
ticular  combination . 

The  following  points  should  be  noted: 

•  This  method  allows  the  analyst  to  focus  on  the  factors  of  inter¬ 
est  and  to  investigate  the  relationships  between  those  factors; 
other  factors,  equally  valid,  can  be  left  to  the  background. 

This  means,  however,  that  the  relationship  between  those  back¬ 
ground  factors  and  the  factors  under  investigation  is  unknown. 

In  particular,  for  man-machine  systems,  it  leaves  unanswered  the 
question,  to  what  extent  is  the  outcome  the  result  of  the  given 
levels  of  the  given  factors  (the  capabilities  of  the  systems  the 
operators  are  working  with),  and  to  what  extent  is  the  outcome 
the  result  of  the  operators  compensating  (with  informal,  perhaps 
ad  hoc  procedures)  for  the  shortcomings  of  the  systems  with 
which  they  work. 

•  This  method  allows  the  analyst  to  investigate  relationships 
between  factors  that  would  ordinarily  be  thought  incommensur¬ 
able.  This  is  particularly  a  problem  in  C2  systems  involving 
complex  man-machine  interactions. 

•  STF  is  not  for  everything.  In  particular,  if  adequate  quanti¬ 
tative  analysis  of  a  problem  already  exists,  then  STF  will  not 
improve  on  this  existing  analysis.  We  feel  that  this  is  the 
case  with  communications  systems,  adequate  analysis  of  connec¬ 
tivity  already  exists,  so  nothing  would  be  gained  ty  applying 
STF  to  analyzing  communications  systems. 

•  Building  a  model  with  this  method  involves  the  concatenation  of 
subtrees  into  a  larger  tree.  For  example,  the  outcome  of  the 
preceding  tree,  Completeness  of  MWC  Information,  is  a  factor  in 
the  tree  in  Fig.  2. 


%  Compute 


Availability 


Figure  2.  Node  for  questionnaire  111 
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Because  the  questionnaires  for  the  different  trees  may  be 
administered  to  different  groups  of  experts,  it  is  vitally 
important  that  the  definitions  of  the  factors  and  the  measures 
be  understood  consistently  by  all  groups. 


A  PLANNING  DECISION  AID  FOR  VANGUARD  ANALYSTS 
Goals 


Practical  use  by  ESD  staff  analysts  is  an  important  quality  for  any  tool 
that  purports  to  help  out  in  the  C3I  Development  Planning  job.  As  part  of  our 
Task  4  plan  we  set  out  to  demonstrate  that  results  of  STF  analysis  could  be 
packaged  for  use  by  Vanguard  analysts.  Our  objectives  were: 

•  to  show  examples  of  program  analysis  and  program  comparison: 
the  tool  should  provide  a  measure  of  the  contribution  to  base¬ 
line  capability  for  each  program  or  program  combination  under 
consideration, 

o  to  Show  how  technology  guidance  can  derive  from  Vanguard  analy¬ 
sis:  the  tool  should  indicate,  through  sensitivity  analysis, 

the  capabilities  that  most  impact  the  overall  mission  value* 

•-  to  demonstrate  easy  user  interface  concepts:  the  tool  should  be 
small  and  portable  and  directly  usable  by  the  Vanguard  analyst 
with  minimal  effort;  hopefully,  it  should  be  reasonably  inter¬ 
esting  to  work  with. 
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3.  EXPERIMENTAL  TEST  OF  STF  APPLICABILITY 


CHOICE  OF  VANGUARD  STRATEGIC  C2  MISSION 

Several  simple  facts  led  to  the  decision  to  test  STF  with  the  Strategic 
Defense  C2  mission. 

•  Analysts  at  ALPHATECH  were  familiar  with  the  mission.  We  real¬ 
ized  that  this  would  be  helpful;  in  fact,  we  discovered  that 
familiarity  with  the  mission  area  is  crucial  to  the  analyst 
pursuing  the  investigation. 

•  ESD  had  good  contacts  at  the  commands.  This  was  even  more 
important  than  we  had  expected. 

o  There  were  not  as  many  strategic  programs  as  in  some  other 

mission  areas,  so  the  job  did  not  appear  overwhelming.  In  fact 
we  ended  up  looking  at  only  a  few  programs.  Also,  this  was  a 
mission  area  in  which  all  the  programs  under  consideration  con¬ 
tributed  ultimately  to  a  common  goal,  which  provided  a  means  of 
comparison  of  these  programs. 

We  focused  on  information  flow  from  the  MWC  to  the  NORAD  CP  to  SAC  and  to 
the  NCA,  and  on  programs  serving  this  flow.  Practical  limits  to  the  resources 
available  for  analysis  and  data  collection  left  us  with  a  reduced  tree  and 
only  a  few  programs  that  it  could  cover.  The  program  comparison  and  analysis 
was  therefore  only  a  demonstration,  not  a  full  scale  test  of  how  STF  could  be 
used  for  Vanguard.  Appendix  A  shows  the  complete  tree  originally  devised, 
and  the  reduced  tree.  In  particular,  the  demonstration  analysis  includes 
most  of  the  the  NORAD  Command  Center  programs,  but  excludes  those  from  SAC. 

It  excludes  communications  systems  for  reasons  discussed  above:  we  can  draw 
better  quantitative  results  from  existing  analytic  studies,  so  we  felt  that 
applying  STF  was  inappropriate.  We  excluded  sensors  and  Air  Defense  systems 
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because  they  belong  to  a  different  mission.  We  also  excluded  mobile  systems 
because  they  made  no  contribution  to  our  scenario,  in  which  post-attack  Force 
Management  was  not  covered.  Similarly  we  excluded  SPADOC  systems  because 
SPADOC  does  not  feed  Strategic  Defense  at  SAC. 

SETTING  UP  STF 
Initial  Visits 

We  began  our  data  gathering  with  initial  visits  to  SAC  and  NORAD.  In 
these  interviews  we  were  concerned  with  structuring  the  hierarchy,  determining 
appropriate  measures  for  each  of  the  nodes  in  tne  tree,  and  determining  appro¬ 
priate  levels  of  the  measures.  Generally  we  found  that  agreement  could  be 
reached  fairly  quickly  on  the  elements  of  the  hierarchy,  and  their  relative 
structure,  but  the  question  of  measures  was  far  more  difficult. 

Later  developments  in  the  project  convinced  us  that  further  data 
gathering  is  very  necessary,  prior  to  administering  the  questionnaires, 
for  the  following  reasons* *. 

o  These  initial  trips  tended  to  focus  on  current  mission  area 
requirements,  with  future  requirements  less  well  discussed. 

•  Not  all  the  factors  that  were  initially  discussed  were  actually 
relevant  at  the  questionnaire  stage. 

o  Many  of  the  factors  were  very  sensitive  to  the  specific 
scenario  being  used  as  context. 

Trees 

As  expected,  in  the  tree  we  developed  in  discussions  with  MAJCOM  per¬ 
sonnel,  the  SAC  Strategic  Offense  Mission  has  three  major  components:  Force 
Warning,  the  survival  options  available;  Force  Direction,  the  offensive 
response;  and  Force  Management,  fine-tuning,  retargetting,  and  R3.  These 
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three  branches  are  not  simply  a  task  breakdown  of  the  C2  mission;  they 
represent  specific  information  th<  t  the  C2  mission  provides  to  the  Strategic 
Offense  mission. 

We  found  that  there  were  three  major  NORAD  information  products  provided 
to  SAC:  Missile  Warning  Data,  the  stream  of  processed  and  summary  data  that 
is  provided  from  the  Missile  Warning  Center  (MWC);  CINCNORAD's  Attack  Assess¬ 
ment,  provided  at  *  minutes  after  each  event  warning,  and  NORAD's  Attack 
Characterization,  again  a  stream  of  data  from  the  NORAD  CP.  Each  of  these 
products  feeds  the  next,  as  well  as  feeding  SAC.  The  relationships  between 
these  can  be  pictured  as  in  Fig.  3. 


'  NORAO  ' 

Attack 


CINCNORAD 


Characterization 
v  / 


I  Missile 
j  Warning 
i  Information 


Figure  3.  Flow  of  Information,  NORAD  to  SAC 
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Formulating  Measures 


As  mentioned  above,  it  was  difficult  to  define  appropriate  measures  for 
many  nodes  of  the  tree.  The  task  was  complicated  because  we  were  trying  not 
only  to  model  the  existing  mission,  but  also  to  gain  insights  into  systems 
that  would  best  suit  future  mission  requirements.  We  had  anticipated  a  need 
to  define  and  discuss  alternative  future  technologies.  We  also  discovered 
that  operational  personnel  were  not  necessarily  familiar  with  future  mission 
requirements,  and  found  it  necessary  to  involve  planning  personnel  from  the 
relevant  offices  to  define  and  discuss  possible  future  mission  requirements. 
The  STF  method  stresses  the  need  to  deal  with  operational  personnel;  however, 
we  found  it  necessary  to  deal  with  technologies  and  requirements  outside  of 
the  day-to-day  experience  of  these  personnel. 

Of  the  measures  that  appeared  the  most  useful,  the  hardest  to  explain 
was  the  concept  of  completeness  of  information.  We  also  used  delay,  rates 
and  qualitative  measures  of  the  man-machine  interface.  Appendix  B  gives  the 
definitions  we  used  and  Appendix  C  contains  a  separate  discussion  of  the 
concept  of  Completeness  of  Information. 

Scenarios 

In  the  design  of  our  experiment  we  tried  to  be  scenario  independent, 
but  in  some  contexts  we  found  that  we  had  to  tie  questionnaires  to  concrete 
examples  to  provide  a  common  basis  for  discussion.  We  devised  a  low  load 
scenario  evolving  to  a  high  load  scenario  as  background  and  for  use  when  such 
discussions  became  necessary.  The  low  load  scenario  was  meant  to  be 
ambiguous;  additional  information  from  several  sources  would  be  needed  to 
assess  the  situation  with  confidence.  This  would  stress  the  Integration  and 
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Assessment  (fusion)  activities  in  the  NORAD  Command  Post.  The  high  load 
portion  of  the  scenario,  on  the  other  hand,  was  chosen  to  stress  the  event 
processing  activities  of  the  Missile  Warning  Center. 

However,  we  eventually  discovered  (at  SAC)  that  the  employment  of  strat¬ 
egic  offensive  forces  was  quite  dependent  upon  nuances  of  the  scenario  and  for 
our  particular  example  CINCNORAD’s  Assessment,  though  normally  an  important 
input,  was  not  a  significant  factor.  Because  requirements  and  decisions  in 
this  mission  are  so  sensitive  to  scenario,  and  because  system  performance  is 
also  scenario  dependent  (i.e.,  interactions  with  threat,  environment  and  other 
sources  of  stress)  we  determined  that  a  more  elaborate  scheme  than  the  one  we 
tried  is  needed  for  including  scenario  effects  in  STF. 

Collecting  Acquisition  Program  Data 

Inputs  at  the  bottom  of  the  tree  describe  the  capabilities  of  systems 
that  are  produced  by  acquisition  programs.  In  the  Vanguard  application  of 
STF  we  measure  program  contribution  and  value  to  the  mission  by  these  system 
capabilities . 

For  most  capabilities  the  contribution  of  a  program  (or  a  collection  of 
programs)  can  immediately  be  estimated  from  basic  information  about  the  sys- 
tem(s)  that  are  being  acquired  and  deployed.  However,  some  system  capabili¬ 
ties  are  defined  by  operationally  oriented  measures  of  performance.  Such 
operationally  oriented  measures  include:  clarity  of  display,  user  friend¬ 
liness,  data  base  query  capability,  sophistication  of  fusion  data  processing, 
and  data  processing  capacity. 

These  capability  measures  depend  on  several  system  attributes,  which  the 
STF  methodolgy  combines  into  single  measure.  In  this  experiment  we  did  not 
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collect  quantitative  data  on  the  transfer  function  describing  how  the  opera¬ 
tional  capability  depends  on  the  attributes.  Instead,  we  chose  a  simple 
weighted  sura  of  attributes  as  a  surrogate  transfer  function. 

The  forms  for  recording  system  attribute  data  are  shown  in  the  charts  in 
Appendix  E.  Weights  for  different  attribute  levels  are  included  on  the  forms. 

For  each  set  of  systems  and  for  each  operationally  oriented  measure  a 
form  is  filled  out  indicating  the  attributes  that  the  system(s)  include.  The 
selected  weights  are  then  added  and  multiplied  by  an  appropriate  scaling  fac¬ 
tor  to  give  a  rough  value  of  the  measure.  More  elaborate  transfer  functions 
can  be  constructed  with  the  same  attribute  data. 

ADAPTING  STF 

Initial  Questionnaire  Design 

We  developed  an  automated  tool  for  generating  questionnaires  in  antici¬ 
pation  of  many  nodes  of  data  collection  and  to  determine  if  STF  could  easily 
be  implemented  on  a  larger  scale.  We  discovered  that  for  many  of  the  nodes, 
a  complete  full-factorial  questionnaire  (asking  all  possible  combinations  of 
levels  of  factors)  would  run  to  several  hundred  questions  and  was  not  practi¬ 
cably  poas4*hle .  Our  initial  questionnaire  design  concentrated  on  asking  more 
questions  of  combinations  of  all  factors  together,  at  the  expense  of  asking 
few  two-way  combinations  of  factors.  We  felt  that  this  would  give  us  the 
best  statistical  fit  of  model  to  data.  Unfortunately  this  masked  the  two-way 
interactions  in  which  we  were  also  interested. 

For  our  second  round  of  visits  we  adopted  a  technique  called  Central 
Composite  design.  This  is  an  efficient  technique  for  choosing  a  minimum 
number  of  questions  posing  combinations  of  all  factors;  it  allowed  us  to 
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ask  a  complete  set  of  two-way  combinations  of  factors ,  and  did  indeed  give 
us  better  insights  into  interactions  between  factors. 

We  believe  that  future  STF  efforts  should  utilize  general  purpose  tools, 
such  as  we  developed,  for  generating  questionnaires,  recording  responses,  and 
doing  data  reduction.  This  means  that  the  "housekeeping**  involved  in  an  STF 
effort  can  be  greatly  reduced. 

What  cannot  be  reduced,  and  we  stress  this,  is  work  with  the  respondents. 
The  originators  of  this  method  (Rand  publication  R-3021-AF)  state  that  three 
visits  are  necessary:  a  first  to  develop  the  trees  and  measures,  a  second  to 
validate  the  tree  and  administer  a  small  dry  run  of  the  questionnaire,  and  a 
final  visit  to  administer  the  full  questionnaire.  Originally  we  questioned 
the  necessity  of  the  second  visit.  We  now  not  only  agree  that  the  second 
visit  is  necessary,  but  we  feel  an  additional  visit,  prior  to  these  three,  may 
be  necessary  to  locate  the  "experts."  Because  future  mission  requirements  and 
future  system  capabilities  must  be  judged,  it  is  not  necessarily  true  that 
operational  personnel  doing  the  jcb  today  are  the  best  respondents.  We  found 
that  personnel  with  operational  experience  who  also  had  recent  experience  in 
the  planning  field  were  most  helpful.  (This  prior  visit  would  also  provide  an 
opportunity  to  explain  and  "sell"  STF.) 

SETTING  UP  THE  ANALYST'S  DECISION  AID 

We  chose  the  Macintosh  personal  computer  as  the  PC  for  implementation  of 
the  prototype  decision  aid  because  it  offered  easy  access  to  interface  options 
such  as:  windows,  mouse,  dialog,  pushbuttons,  menus,  graphics,  and  the  like. 
We  made  a  deliberate  attempt  to  sample  several  different  interface  techniques 
to  show  ESD  the  kind  of  things  they  should  ask  for  and  expect  in  future  data 
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processing  tools.  As  a  prototype,  we  expect  that  the  decision  aid  will  not 
have  operational  use  and  will  provide  concepts  for,  but  not  be,  an  operational 
baseline  for  subsequent  applications. 
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A .  RESULTS 


This  section  summarizes  the  results  of  the  experimental  test  of  the  STF 
method  applied  to  Vanguard  ana  Lysis.  The  test  was  successful,  but  we  did  not 
cover  as  much  ground  as  we  had  wished.  We  learned  a  lot  about  the  relation¬ 
ships  among  operational  requirements,  system  capabilities  and  program  evalua¬ 
tion.  In  addition,  as  was  planned,  we  did  implement  and  install  a  model 
on  a  personal  computer.  Some  of  what  we  learned  is  applicable  to  Vanguard 
analysis,  some  applies  more  broadly  to  Development  Planning  and  System 
Acquisition. 

THE  CAUSAL  TREES 

We  believe  that  the  causal  tree  developed  in  this  project  has  value  even 
apart  from  the  software  tool  in  which  it  is  embedded.  It  offers  an  alterna¬ 
tive  and  complementary  view  of  the  mission  area  from  that  mandated  in  AFSCP 
80-3  and  used  in  Vanguard  today. 

Since  the  trees  delineate  the  system  capabilities  that  operators  them¬ 
selves  find  important  in  their  work,  the  trees  can  be  used  by  Vanguard  ana¬ 
lysts  as  a  first  step  in  program  evaluation.  In  particular,  they  show  what 
capabilities  need  to  be  present  together  to  accomplish  a  particular  function. 
Similarly  they  should  be  helpful  to  analysts  doing  Technology  Planning  and 
System  Development. 
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DATA  COLLECTION  EFFORT 

We  encountered  several  difficulties  in  data  collection.  We  have  already 
mentioned  the  difficulties  in  defining  measures;  these  were  sometimes  diffi¬ 
cult  to  explain.  Clearly,  from  the  responses  on  the  questionnaires,  we  can 
see  that  our  understanding  of  some  of  these  measures  was  not  the  same  as  the 
respondents’.  For  example,  the  relationship  between  completeness  of  infor¬ 
mation  and  time:  Is  completeness  something  that  must  increase  with  time  (as 
you  get  more  information  processed)?  Or  is  it  something  that  can  be  expected 
to  decrease  (because  the  amount  of  information  you  need  increases  relative  to 
what  you  have  —  or  —  because  the  amount  of  information  that  is  available  in 
Mthe  real  world'*  is  greater  than  you  can  process)?  Another  difficulty, 
already  mentioned,  was  the  problem  of  asking  operational  personnel,  whose 
experience  and  overwhelming  concern  is  with  the  current  mission  requirements, 
to  make  judgments  about  future  mission  requirements*  Because  we  underesti¬ 
mated  the  amount  of  preparation  needed  we  sometimes  found  ourselves  rede¬ 
signing  questionnaires  and  refining  definitions  and  scenarios  on  the  spot* 
Lastly,  there  were  practical  difficulties  involved  in  locating  appropriate 
personnel  and  getting  them  together  in  the  same  place  at  the  same  time  for 
the  amount  of  time  we  needed  —  typically  at  least  half  a  day,  sometimes  an 
entire  day.  For  no  questionnaire  did  we  have  more  than  three  respondents. 

The  implication  of  the  above  is  that  we  feel  it  unlikely  that  a  Vanguard 
analyst  could  do  an  STF  analysis  of  his  mission  area  unaided.  It  requires 
considerable  expertise  in  the  operational  aspects  of  the  mission  area  as  well 
as  considerable  expertise  in  STF,  and  a  lot  of  preparation. 

In  addition  to  collecting  mission  data  from  the  Major  Commands,  we  needed 
to  collect  program  data  from  ESD.  Since  the  number  of  programs  that  our 
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reduced  mission  area  coverage  could  include  was  very  limited,  the  latter  was 
rather  cursory.  We  reviewed  the  data  calls  and  relied  on  our  knowledge  of 
the  programs  we  felt  we  could  realistically  include  in  the  demonstration  to 
complete  the  program  contribution  sheets. 

STF  ANALYSIS 

Reducing  the  questionnaire  responses  to  transfer  functions  requires  con¬ 
siderable  effort.  We  used  a  general  data  reduction  tool  that  uses  statistical 
techniques  similar  to  curve  fitting.  Had  we  been  able  to  analyze  the  entire 
mission  area  we  would  have  done  far  more  questionnaires.  We  aould  not  have 
accomplished  it  without  this  tool. 

As  it  was,  for  at  least  one  of  our  questionnaires  the  best  transfer  func¬ 
tion  we  could  devise  has  a  larger  chi-square  measure  than  we  would  like.  We 
feel  that  the  questionnaire  design  in  this  case  was  poor. 

Some  of  the  interactions  displayed  by  the  quesionnaire  data  made  obvious 
intuitive  sense.  The  questionnaire  covering  fusion  (information  integration 
and  assessment)  showed  a  relationship  between  data  processing  capacity  and 
display:  the  more  data  processing  capacity,  the  more  important  to  have 

sophisticated  display  available;  the  relationship  is  as  shown  in  Fig.  4.  As 
data  processing  capacity  increases  we  see  that  increased  levels  of  display 
make  more  of  a  difference  to  the  outcome.  The  result  is  that  the  plots  spread 
out  in  a  fan  shape;  if  there  were  no  interaction  the  plots  would  be  parallel. 

In  other  cases,  relationships  that  we  expected  did  not  appear.  On  the 
same  questionnaire  we  expected  a  similar  relationship  between  access  to  other 
data  and  display:  we  expected  that  the  more  non-MWC  data  was  available,  the 
more  important  display  would  be.  This  did  not  appear  in  the  data;  see  Fig.  5. 
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The  plots  are  (nearly)  parallel  because  the  Increase  In  display  capability 
makes  the  same  difference  In  the  outcome,  regardless  of  the  level  of  complete¬ 
ness  of  other  information.  Perhaps  there  is  little  interaction  between  these 
factors.  We  suspect  that  we  asked  the  wrong  questions  or  that  we  asked  them 
in  the  wrong  way.  In  this  Instance  we  suspect  that  we  did  not  make  clear  that 
other  data  was  not  something  that  would  just  somehow  "be  there;”  i.e.,  that 
other  data  would  also  need  to  be  displayed. 

Usefulness  of  Results 

We  are  convinced  of  the  potential  value  of  the  STF  method.  It  indeed 
provides  an  excellent  way  of  getting  an  approximation  of  the  nature  of  complex 
Interactions  between  multiple  factors  affecting  an  outcome.  The  nature  of  the 
complete  STF  -model  is  such  that,  if  more  precise  analytic  relationships  become 
available,  it  is  easy  to  substitute  these  for  the  transfer  functions  developed 
from  the  questionnaires. 

We  caution,  however,  that  the  results  of  this  project  will  not  lead 
directly  to  a  Vanguard  analyst’s  support  tool.  We  were  unable  to  cover  an 
entire  mission  area  with  the  time  and  effort  available  and  we  still  feel  the 
question  of  appropriate  respondents  for  an  evolving  mission  area  remains  open- 

TOOL  DEVELOPMENT 

In  Task  3  of  the  project  we  surveyed  available  computer  models  of  C2 
systems.  We  determined  that  few  were  applicable  to  the  problem  Vanguard 
addresses  and  that  none  of  these  was  practical.  Often,  these  applicable 
models  were  not  practical  because  they  lacked  a  model  environment.  The 
relationship  between  a  model  and  Its  environment  is  shown  in  Fig.  6. 
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Figure  6.  Model  and  Environment 

A  model  environment  includes  a  user  interface,  a  case  generator,  a  data 
checker  and  an  explainer.  A  user  interface  ma<es  it  easy  to  use  the  other 
parts  of  the  environment,  as  well  as  running  the  model  itself.  A  case  genera¬ 
tor  makes  it  easy  to  change  selected  variables  of  the  model  and  to  create  new 
data  sets.  A  data  checker  ensures  that  the  data  prepared  by  the  case  genera¬ 
tor  fits  the  requirements  of  the  model,  and  if  not,  indicates  why  not.  Final¬ 
ly,  an  explainer  both  interprets  the  model  results  and  explains  how  they  were 
developed. 

Our  prototype,  the  Vanguard  Analysts  Support  Tool  (VAST)  incorporates 
much  of  the  necessary  model  environment.  Figures  7  through  12  show  screens 
which  VAST  uses  to  interface  with  the  analyst.  The  interface  is  graphic  where 
possible,  and  relies  on  the  use  of  windows,  buttons,  and  the  mouse,  as  well 
as  default  choices,  to  reduce  analyst  effort  to  a  minimum.  Figure  7  shows  the 
opening  screen;  from  this  screen  the  analyst  selects  an  item  from  one  of  the 

menus.  If  he  wishes  to  create  a  new  data  set,  he  is  shown  the  screen  in 

Fig.  8-  The  upper  window  allows  the  analyst  to  name  the  data  set  and  the 
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Figure  8. 4  Creating  a  New  Data  Set 
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Figure  9.  Basing  the  New  Data  Set  on  an  Existing  Data  Set 
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Figure  11.  Information  Screen 


R-2919 
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lower  window  allows  him  to  base  his  new  data  set  on  an  existing  data  set. 
Figure  9  shows  the  screen  for  selecting  the  existing  data  set.  Having 
selected  a  data  set.  Fig.  10  shows  the  top  level  of  the  tree  for  this  data 
set.  The  UP  (up)  and  DN  (down)  buttons  allow  the  analyst  to  view  other  levels 
of  the  tree.  The  ?  (information)  button  provides  information  about  each  node 
and  the  IN  (enter)  button  allows  the  analyst  to  change  the  value  at  any  node. 
The  RECALCULATE  button  initiates  a  recalculation  of  all  values;  when  completed 
the  screen  appears  with  new  values  as  calculated.  Figure  11  shows  the  result 
of  selecting  ?  for  node  1  and  Fig.  12  shows  the  result  of  selecting  IN  for 
node  1.  Note  that  in  place  of  an  explicit  data  checker,  we  restrict  the 
choice  of  values  for  each  node  to  a  valid  range. 

The  Analyst’s  Decision  Aid,  VAST,  uses  as  a  basic  data  structure  a  Data 
Set  that  represents  a  level  of  capability  from  groups  of  programs.  We  have 
included  a  Data  Set  that  represents  Minimal  Levels,  and  a  Baseline  Data  Set. 

It  is  our  estimate  that  the  baseline  capability  represents,  roughly,  the 
capabilities  presently  available  in  the  NORAD  MWC  and  CP  systems.  For  each 
of  the  programs  considered  in  the  demonstration  (SCIS,  AFWIS,  etc),  there  is 
a  data  set  which  represents  the  baseline  plus  the  added  capability  from  the 
program.  We  designed  a  contribution  form  which  facilitates  entry  of  data 
into  the  data  set;  for  the  qualitative  capabilities  we  designed  checklists  of 
system  attributes  which  will  aid  in  estimating  the  level  of  capability  that 
the  program  provides.  In  this  initial  effort  it  is  not  possible  to  compare 
combinations  of  programs. 

We  believe  the  software  tool  developed  on  the  PC  demonstrates  that  suc¬ 
cessful  incorporation  of  STF  concepts  in  a  tool  that  the  Vanguard  analyst  can 
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use  directly  is  feasible.  The  prototype  took  approximately  four  person- 
months  to  develop.  We  are  still  unsatisfied  with  the  way  in  which  program 
data  is  incorporated  and  suggest  extensions  for  a  second  edition  that  would 
involve  automating  the  checklists  and  programs  contribution  sheets.  A  next 
edition  could  take  six  person-months  to  develop.  This  is  still  modest  as 
software  costs  go.  This  tool  is  an  example  of  the  simple,  PC-oriented  tools 
that  we  recommended  in  Task  two. 

VANGUARD  PROGRAM  EVALUATION 

The  program  evaluation  that  the  prototype  software  tool  provides  must  be 
used  very  carefully  if  at  all.  Program  comparisons  are  available  but  should 
be  interpreted  carefully.  Given  the  data  content  of  the  tool,  the  prioritiza¬ 
tion  of  programs  is  not  significant.  The  sensitivity  analysis  that  the  tool 
provides  is  a  useful  concept  but  again,  is  probably  not  significant  due  to  the 
lack  of  rigorous  data. 
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5.  CONCLUSIONS 


SUMMARY 

The  immediate  problem  we  faced  in  applying  STF  to  ESD's  Vanguard  mission 
area  analysis  was  that  of  separating  the  process  of  Command  and  Control, 
leadership,  if  you  prefer,  from  the  evaluation  of  Command  and  Control  systems. 
Our  primary  conclusion  is  that  this  can  be  done;  our  solution  was  to  approach 
C2  system  as  information  systems  whose  purpose  is  decision  support.  Implicit 
in  this  approach  is  the  assumption  that  the  better  the  decision  support  avail¬ 
able,  the  better  the  decision  that  is  made.  Beyond  this,  we  did  not  attempt 
to  analyze  how  decisions  are  made  by  commanders. 

Thus  we  focused  on  information  generation  and  transformation  from  the 
sensors  through  to  SAC  CP,  and  attempted  to  determine  how  capabilities  of 
information  systems  affected  the  quality  of  the  information  available:  time¬ 
liness,  accuracy,  completeness,  relevance  (priority),  etc. 

We  conclude  that  STF  can  be  an  effective  modeling  tool  for  mission  area 
analysis  and  program  evaluation.  It  is  not  a  panacea  for  Vanguard.  STF  needs 
to  be  done  carefully,  and  with  a  lot  of  preparation.  It  is  expensive. 

Apart  from  STF,  we  feel  the  prototype  software  tool  developed  on  the  PC 
demonstrates  that  tools  can  be  inexpensively  developed  that  incorporate 
relatively  simple  yet  powerful  ideas,  and  that  the  analyst  can  use  directly 
without  a  software  expert  as  intermediary.  As  we  stressed  in  our  Task  three 
report,  the  user  friendly  interface  is  highly  important. 
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ABOUT  STF  APPLICABILITY 


We  leave  open  the  question  of  whether  STF  should  be  applied  to  further 
Vanguard  analysis,  even  though  we  are  convinced  of  the  basic  merit  of  the 
method.  We  believe  it  is  applicable  to  other  analyses.  The  trees  are  a  use¬ 
ful  tool  for  requirements  definition;  they  provide  a  common  basis  for  discus¬ 
sion  between  operational  personnel  and  planners  as  to  what  is  important  and 
what  is  related  to  what.  The  capability  sensitivities  and  interactions  are 
useful  for  technology  planning:  they  tell  technology  planners  what  capabili¬ 
ties  Mgo  together”  and  where  tradeoffs  can  be  made. 

A  major  limitation  of  the  STF  method  is  that  it  forces  measures  into  one 
dimension.  Operators  might  feel  that  MWC  information,  for  example,  should  be 
measured  both  by  completeness  and  by  timeliness.  As  now  defined,  STF  forces 
the  analyst  to  choose  one  measure,  or  to  invent  some  surrogate  measure  that 
somehow  incorporates  both  completeness  and  timeliness.  This  surrogate  measure 
may  not  be  intuitively  obvious  to  personnel  who  are  being  asked  to  make  judg¬ 
ments  about  it. 

In  some  cases  the  precision  and  accuracy  that  STF  requires  is  not  needed: 
for  some  program  decisions  qualitative  relationships  would  be  just  as  useful. 
But  on  the  questionnaire  the  respondent  is  required  to  make  a  precise  numeric 
judgment;  he  cannot  respond  with  a  range  of  values,  or  a  qualitative  judgment. 

We  have  discussed  elsewhere  the  lack  of  expertise  in  future  technologies 
and  evolving  mission  requirements. 

ABOUT  ANALYST  DECISION  AIDS 

We  still  believe  in  analyst  decision  aids  that  are  PC-based,  useful  and 
inexpensive  to  develop.  The  ideas  and  data  content  that  inform  these  decision 
aids,  however,  may  not  be  cheap. 
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ABOUT  RESOURCE  COSTS 


The  question  arises:  What  resources  would  be  required  to  do  the 
Strategic  C2  Vanguard  sub-mission  area  in  another  Vanguard  cycle,  given  the 
preceding  conclusions? 

These  are  the  assumptions  tha^  were  made  in  arriving  at  the  following 

estimates : 


1 •  The  size  of  the  mission  area  hierarchy  will  not  vary  much  from 
the  tree  devised  for  this  effort:  there  will  be  about  90  nodes 
and  25  questionnaires. 

2.  The  number  of  programs  will  continue  to  be  about  30. 

3.  The  data  reduction  software  (to  convert  questionnaire  results  to 
transfer  functions)  is  satisfactory.  This  software  is  written 
in  Fortran  to  run  on  a  VAX  11/750.  It  required  eight  person- 
weeks  to  write;  it  would  probably  require  three  person-days  to 
convert  it  to  run  on  another  VAX  system,  somewhat  more  to  run  on 
some  other  system. 

4.  All  BASIC  programs  should  be  rewritten  in  a  more  structured 
language  on  a  common  micro-computer: 


STF 10: 

data  entry  for  nodes 

5 

days 

STF20: 

print  questionnaires  using  Central 
Composition  design 

5 

days 

STF30: 

data  entry  for  questionnaires 

5 

days 

VAST: 

Vanguard  Analyst  Support  Tool 

50 

days 

(All  the  above  estimates  include  time  required  for  design 
and  documentation.) 

5.  Three  visits  will  be  required  for  each  of  the  major  commands, 

SAC  and  NORAD.  This  assumes  that  a  preliminary  trip  to  famil¬ 
iarize  the  commands  with  STF  will  not  be  necessary,  and  that  the 
required  group  of  experts  can  be  located,  and  their  cooperation 
agreed  to,  either  by  telephone  or  by  local  cooperation.  It  also 
assumes  that  it  may  not  be  possible  to  make  the  first,  second 
and  third  visits  to  SAC  and  to  NORAD  on  the  same  trip,  because 
of  the  difficulties  of  getting  the  required  experts  together. 

On  the  other  hand  it  does  assume  that  all  the  expert  groups  at 
one  command  can  be  interviewed  on  the  same  trip. 
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With  the  above  assumptions,  the  following  are  estimates  of  person-days  of 


effort  required  for  each  of  the  tasks  shown. 

0.  Preliminary:  Software  —  see  above 

1.  Structure  trees  and  devise  measures 

two  people  at  1/2  day  per  questionnaire* 

-  from  the  major  commands,  three  people  at  1/2  day 
per  questionnaire 

2.  Check  overall  tree  structure,  node  data  entry  and 
print  questionnaires 

3.  Follow-up  visit 

-  two  people  at  1/4  day  per  questionnaire 

from  the  major  commands,  three  people  at  1/4 
day  per  questionnaire 

4.  Preliminary  analysis  at  1/4  day  per  questionnaire 

5.  Fix  any  problems 

6.  Final  visit 

two  people  at  3  questionnaires  per  day 

-  from  the  major  commands,  three  people  at 
3  questionnaires  per  day 

7.  Data  entry  of  results 

8.  Data  reduction  at  1/2  day  per  questionnaire 
9-  Get  results  into  VAST  and  check 

10.  Analyze  programs  at  1/2  day  each 

(This  assumes  some  familiarity  with  the  programs.) 

11.  Write  reports,  prepare  briefings 


*The  estimate  of  four  hours  per  questionnaire  is  an  average, 
less,  others  more.  For  the  follow-up  and  final  visits  the 
hours  per  questionnaire  assumes  that  not  all  questionnaires 
be  discussed:  many  would  be  similar  to  one  another. 


65  days 

25  days 
38  days 

5  days 

12  days 
18  days 

6  days 
5  days 

10  days 
15  days 

2  days 
12  days 
10  days 
15  days 

10  days 

some  would  take 
estimate  of  two 
would  need  to 
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Other  charges: 


1.  A  micro-computer  $4,000 

2.  12  round-trips  to  the  West  at  $500  $6,000 

3.  Other  travel  at  $100  per  day  $4,800 

4.  VAX  time  at  1  hour  per  questionnaire  $7,500 


ABOUT  REVISION  OF  THE  MODEL  IN  SUBSEQUENT  YEARS 

The  following  estimates  assume  that  the  higher  levels  of  the  tree  repre¬ 
sent  organizational  arrangements  which  are  not  likely  to  change  from  year  to 
year,  but  the  lower  levels  of  the  tree  represent  local  operational  procedures 
and  the  systems  which  support  those,  and  these  are  likely  to  change.  We 
assume  therefore  that  40  nodes,  10  questionnaires,  might  change  and  that  the 
changes  could  be  covered  in  two  visits. 

# 

1.  Preliminary  visit 

-  two  people  at  1/2  day  per  questionnaire  10  days 

-  from  the  major  commands,  three  people  at  1/2  day  15  days 

per  questionnaire 

2.  Final  visit 

-  ESD  effort  (same  as  above)  10  days 

-  major  command  effort  (same  as  above)  15  days 


3.  Data  Entry  3  days 

4.  Data  Reduction  3  days 

5.  Modify  VAST  5  days 

Other  charges: 

1.  4  round  trips  to  the  West  $2,000 

2.  travel  at  $100  per  day  $2,000 

3.  VAX  time  -  ten  hours  $3,000 


39 


6. 


RECOMMENDATIONS 


A.  ABOUT  VANGUARD  DATA  COLLECTION 

1.  Program  data  should  be  collected  in  a  more  structured  way. 

We  recommend  developing  checklists  for  collecting  program  data  and  incor¬ 
porating  these  into  the  Vanguard  data  call.  These  should  also  be  incor¬ 
porated  into  the  automated  data  base  now  being  developed  for  Vanguard. 
Eventually  much  of  the  program  data  collection  could  be  automated.  As  we 
have  recommended  in  previous  tasks,  data  collection  for  Vanguard  should 
not  be  an  isolated  effort:  the  evolving  body  of  program  data  should  be¬ 
come  a  resource  for  other  planning  activities;  thus  it  should  be  consis¬ 
tent  with  data  collection  requirements  for  other  planning  tasks. 

B.  ABOUT  QUANTITATIVE  TOOLS  FOR  VANGUARD 

1.  As  we  discussed  in  our  task  3  report,  we  do  not  recommend  that  ESD  acquire 
large  computer  models  for  Vanguard.  Large  computer  models  are  expensive f 
the  few  available  models  of  command  and  control  in  Vanguard  mission  areas 
are  either  difficult  to  use  or  do  not  cover  the  mission  area  adequately. 

2.  This  task  tested  the  Subjective  Transfer  Function  method  for  constructing 
and  quantifying  a  model  of  one  Vanguard  sub-mission  area.  Our  summary 
conclusion  is  that  this  method  will  work,  it  is  still  less  expensive  than 
a  large  computer  model,  but  it  is  not  inexpensive.  We  estimate  that,  for 
the  complete  Strategic  C2  Vanguard  sub-mission  area,  this  method  would 
require  183  person-days  of  effort,  totaling  approximately  one  person- 
year.  If  we  estimate  one  person  year  at  $100K,  this  is  probably  one 
quarter  —  or  less  —  of  what  a  computer  model  would  cost. 

C.  ABOUT  STRUCTURAL  MODELING 

1.  Do  more  structural  modeling  of  sub-mission  areas  using  causal  hierarchies. 

We  found  that  the  construction  of  causal  hierarchies  (trees)  was  an 
excellent  vehicle  for  discussion  of  mission  area  requirements.  As  with 
any  good  modelling  method,  it  provides  a  way  to  decompose  the  problem 
into  subproblems  (nodes  of  the  tree).  This  allows  the  analyst  to  discuss 
different  parts  of  the  problem  with  different  groups  of  operational  and 
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development  personnel.  Even  if  transfer  functions  are  not  derived,  the 
resulting  representation  of  the  mission  provides  a  basis  for  discussion 
that  is  meaningful  to  analyst,  operational  personnel  and  system  devel¬ 
opers.  For  these  reasons,  ESD/XR  should  also  consider  extending  struc¬ 
tural  modeling  using  causal  hierarchies  to  other  areas  of  development 
planning . 

2.  Develop  structural  models  of  other  development  planning  activities. 

Our  experience  with  modeling  the  Vanguard  process  using  IDEF  and  with 
modeling  the  strategic  defense  mission  using  causal  hierarchies,  has 
convinced  us  of  the  value  of  structural  modeling  of  diverse  activities. 
Structural  models  give  people  a  graphic  basis  for  discussion  and  provide 
insights  not  available  from  descriptive  narrative.  Other  development 
planning  activities,  besides  Vanguard,  and  other  modeling  methods  (Petri 
nets,  data  flow  diagrams,  etc.)  should  be  considered. 


D .  ABOUT  STF 

1.  Consider  modifying  the  data  collection  methodology  of  STF  to  adapt  to 
evolving  missions. 

As  discussed  previously,  operational  personnel  currently  working  with 
information  systems  are  not,  initially,  particularly  attuned  to  possible 
future  technologies  or  evolving  mission  requirements.  Consideration 
should  be  given  either  to  locating  more  appropriate  "experts,"  or  quickly 
making  current  operational  personnel  familiar  with  the  required  new  con¬ 
cepts,  or  using  teams  comprised  of  both  operational  and  technological 
experts . 

2.  Consider  simplifications  of  measures  and  ways  to  quantify  them  other  than 
by  deriving  subjective  transfer  functions. 

More  thought  needs  to  be  given  to  the  appropriate  "principles  of  informa¬ 
tion:"  timeliness,  accuracy,  completeness,  relevance,  etc.  —  What  are 
the  appropriate  principles  and  how  should  they  be  measured?  In  particu¬ 
lar,  qualitative  measures  using  range  instead  of  point  estimates  should 
be  considered  where  appropriate. 


E.  ABOUT  DECISION  AIDS 

1.  Insist  that  decision  aids  be  "user-friendly." 

Decision  aids  should  not  only  incorporate  good  ideas,  they  should  also 
embed  good  ideas  in  software  that  is  above  all  self-explanatory;  it  should 
also  be  portable,  keep  data  entry  to  a  minimum  and  be  reasonably  inter¬ 
esting  to  use.  These  are  the  principles  we  have  attempted  to  incorporate 
into  our  prototype,  especially  by  providing  mouse-driven  input  and  inter¬ 
action  and  graphics.  ESD/XR  should  insist  on  decision  aids  that  meet  and 
even  exceed  these  standards. 
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F.  OTHER  RECOMMENDATIONS 


1.  Establish  a  framework  for  future  analytic  modeling  efforts. 

In  task  three  of  this  project  we  determined  that  computer  models  which 
were  both  applicable  to  ESD  needs  and  practical  for  ESD  Vanguard  analysts’ 
use,  did  not  exist.  ESD/XR  should  establish  minimum  standards  for  appli¬ 
cability  and  practicality  of  future  modeling  efforts.  The  latter  should 
include  a  user-friendly  interface,  a  case  generator,  a  data  checker  and 
an  explainer. 


! 
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APPENDIX  A 
TREE  STRUCTURE 

These  five  charts  show  the  tree  as  originally  developed.  The  portions 
of  the  tree  for  which  we  were  not  able  to  collect  data  are  indicated. 
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SAC:  FORCK  WARN1NC 
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NORAD;  MISSILE  WARNING 
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DECOMPOSITION 

OF  NODE  111  *.2111 


NORAD:  ATTACK  WARNING 
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NORAD:  ATTACK  CHARACTKKI ZATl ON 
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DECOMPOSITION  Of  NODE  232 


APPENDIX  B 


DEFINITIONS 
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RUAILABLE  UflflHIHG  IHFORnRTlOH  (nodo  1111)  is  the  missile 
■orning  doto  streoa  generoted  by  the  fllssiie  Horning  Center  (HUC). 
Rvoilobility  is  described  by  the  completeness  of  the  inforaotion  — 
i.e.,  of  the  inforaotion  items  thot  could  be  reody  for  use  ot  o  giuen 
time,  hoa  much  hos  octuolly  been  processed  ond  is  in  foct  reody  for 
the  user.  Specific  inforaotion  (teas  Include:  sensor  doto  (discrete 
event  aessoges,  stotus  aessoges,  suaaory  aessoges);  confidence 
factors;  suaaory  data  generoted  by  the  HUC :  other  inforaotion  items 
if  oppropriote. 

OELIUEREO  URRHIHG  IHFORnRTlOH  to  SRC  (node  111)  is  the  missile 
■orning  doto  streoa  generoted  by  the  HUC  ond  degroded  by 
coaaunicot ion  deloy  ond  quolity  on  its  aoy  to  SRC. 

OELIUEREO  URRHIHG  IHFORnRTlOH  to  HCP  (node  211)  is  the  HUC 
doto  delivered  to  the  HORRO  Coaaond  Post.  Despite  the  close 
proximity  of  the  HUC  ond  the  CP,  explicit  oHoaonce  must  be  mode  for 
the  interaediote  systems  thot  aoke  HUC  inforaotion  products  ovollobie 
to  CIHCHdflflD  ond  his  stoff. 

OELIUERED  CIHCHORRD  ASSESSttEHT  (node  112)  is  the  ossessaent 
produced  in  the  HOARD  Coaaond  Post  ond  delivered,  principolly  by 
voice,  to  the  SRC  Coaaond  Post;  os  degroded  by  deloy  in  setting  up 
the  voice  circuit.  This  ossessaent,  together  with  subsequent 
detoiled  Rttock  Chorocter izot ion  inforaotion,  is  oiso  provided  to  the 
HCR  to  support  response  decisions. 

ABILITY  TO  FUSE  [Integrate  inforaotion  and  assess]  (node 
113)  is  the  copobility  of  the  SRC  Coaaond  Post  system  to  provide 
'complete*  non-missile  ■orning  inforaotion  to  the  Senior  Controller, 
The  system  is  described  by  the  degree  of  completeness  of  thot  other 
inforaotion  ot  porticulor  times  ofter  the  initiol  event  detection 
(alarm). 

The  corresponding  fusion  copobility  for  HOARD  is  captured  by  the 
fivoi loble  Rttock  Uorning  node. 

nUC  DRTR  PROCESS IHG  CAPABILITY  (node  11111)  defines  the 
capacity  of  the  RDP  support  to  the  HUC.  The  measure  Is  hoa  quickly 
aessoges  from  the  sensors  con  be  processed  by  the  system.  There  Is 
cleorly  o  relot ionship  betaeen  hoa  quickly  aessoges  con  orrive 
(Recess  to  Sensor  Doto)  ond  ho»  quickly  they  con  be  hondled. 
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CURREHCY  OF  HUC  SYSTE11  (system  Flexibility)  (node  11111) 
Indicates  ha*  *ell  the  system  con  respond  to  changing  requirements 
for  the  morning  mission.  Typical  changes  include  nem  processing 
algorithms,  nem  and  revised  dlsploy  requirements,  addition  of  ne* 
sensors,  revised  message  sets,  and  changes  In  data  bases.  If  the 
system  cannot  keep  up  *ith  these  evolving  mission  requirements  its 
performance  (completeness  of  information  ovoiloble)  moy  go  do*n. 

CLARITY  OF  OISPLRY  (nodes  11112,  11213,  111)  describes  the 
sophistication  of  the  disploy  system  thot  presents  missile  morning 
ond  other  information  to  the  decision  maker  ond  his  staff.  The  HORRD 
MIC,  the  NORflO  Co»mond  Post,  ond  SRC  Hq  each  hove  their  o»n  disploy 
system. 

-  paper  only  meons  no  outomoted  disploy.  flonuols,  reports 
ond  mritten  nates  ore  ovoiloble.  fionuolly  generated 
viemgrophs  ore  o  half  step  up  from  “paper  only.* 

-  outomoted  tobulor  disploy  meons  thot  oil  information  con  be 
presented  on  group  or  morkstotlon  disploy  devices  os  text  or 
simple  tobies. 

-  bosic  qrophics  display  meons  a  variety  of  chorts, 
histograms,  diagrams,  ond  mops  with  overlays  ore  ovoiloble 
forms  for  portraying  essentially  oil  of  the  information  in 
the  Command  Past.  Hate  that  the  level  of  capability  in  many 
existing  (1985)  command  center  systems  lies  roughly  betmeen 
"outomoted  tabular-  and  “bosic  graphics.* 

-  enhonced.  reol  time,  interactive  graphics  is  the  tap 
quality  display.  Haps  and  chorts  ore  continuously  updated  os 
ne*  data  arrives.  Users  moy  interact  directly  mith  the 
display  (via  menus,  painters,  etc.)  to  request 

di f ferent/detol led  information,  fid  hoc  chorts  moy  be 
generated  quickly. 

ACCESS  TO  SEHSOA  DATA  (node  11113)  is  the  effective  rote  ot  mhich 
Information  arrives  from  the  sensors  ta  the  MIC.  This  is  essentiolly  o 
measure  of  the  communications  systems.  MIC  information  con  be  less 
complete  If  potent iolly  useful  sensor  doto  is  deloyed  (or  lost)  by 
communications  problems. 

ORTA  COnnUMICATIOHS  HUH! LAB  I L 1 TY  (node  111132)  is  the  percent 
af  active  (surviving)  sensors  for  mhich  the  doto  links  to  the  MJC  ore 
marking.  This  mill  af  course  depend  on  mhich  communications  systems 
are  deployed  ond  ha*  each  af  the*  performs  against  potential  threots 
ond  other  scenario-defined  stress. 
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DATA  COnnUHICATIOHS  BAHDU1DTH  (node  111131)  is  the  overall 
description  of  the  capacity  af  the  cai»unlcot ions  systems.  Of  the 
sensors  tbot^aro  connect md  and  sending  stotus  ond_event_ messages  to 
the  nUC,  this  foctor  tells  horn  fost  those  lessoges  can  be  sent  on  the 

automated  data  links* 

UOICE  AUA ILAB ILITY  (node  111133)  Is  a  measure  af  the  avollobilty 
af  voice  communications  mith  a  typical  sensor.  Ualce  is  critical  bath 
as  an  alternative  medium  far  sending  messages  if  automated 
communications  are  degraded  and  as  the  priaary  aeons  of  validating  the 
operational  status  af  the  sensors  and  the  accuracy  af  the  data  being 
sent  ta  the  flUC. 

ACCESS  TO  OTHER  DATA  (node  11211)  describes  haa  tell  the  Attack 
Darning  (Fusion)  function  is  supported  by  systeas  to  retrieve 
interaction  fraa  other  sources  that  are  available  ta  the  decision 
■aker  in  his  caaaand  past.  ["Other"  aeans  other  than  the  aissile 
earning  data  fraa  the  flUC  and  can  include  Information  fraa  other 
aissian  areas  (e.g.  Rir  Defense),  resource  status,  Intel,  trend  data, 
historical  data,  order  af  bottle  and  systea  chororocterist ic 
descriptions,  check  lists  and  procedural  interaction,  etc.]  The 
factor  is  aeasured  by  haa  auch  af  that  interaction  (caap leteness)  can 
be  aaved  fraa  its  source  and  into  the  caaaand  past  In  a  reasonably 
short  time. 

FUSIOH  DATA  PA0CESSIH6  (node  11212)  tells  haa  mell  the  RDP 
systea  supports  the  Fusion  function. 

■no  outoaated  fusion  aeans  that  there  is  no  RDP  support  far 
the  fusion  function.  Fusion  Is  still  passible  mith  manual 
(and  mental)  systems  and  mith  semi-automated  systems  such  as 
CCTU  and  phane  conferences,  [corresponds  ta  value  -  0] 

-limited  peacetime  data  fusion  means  the  RDP  system  supports 
continuous  earning  mith  several  indicators  of  overall 
situation  status.  Uariaus  kinds  af  information  fraa  multiple 
sources  are  combined.  Information  associated  mith  a  single 
missile  event  can  be  handled,  [corresponds  to  value  ■  3) 

-small  scenario  data  fusion  mith  limited  complexity  aeons 
that  information  fraa  a  scenario  aith  a  relatively  seal  I 
number  af  events  can  be  handled,  flultlple  sensor  inputs, 
correlations  among  events,  and  correlation  aith  doto  from 
multiple  a i ss ions/ sources  are  fused  into  a  "big  picture"  af 
the  situation.  But  not  all  relevant  information  fraa  all 
sources  can  be  considered  at  once;  and  not  ail  fusion  Is 
automatically  performed,  considerable  human  intervention  may 
be  necessary  ta  guide  the  evaluation  af  doto.  [value  ■  6] 


-large  complex  scenario  data  fusion  aeons  that  essentially 
all  relevant  Information  from  all  sources. Is  automat  leal ly 
considered  for  o  hlgh-lood  scenario.  The  fusion  results  ore 
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reported  to  decision  makers  os  soon  os  or  even  before  they 
ask  far  thei.  [There  is  af  course  an  assumption  here  that  the 
flOP  system  mill  have  been  pre-programmed  mlth  the  set  af 
quest lans/evaluat Ians  that  decision  makers  are  interested  in 
knaming  about.]  [corresponds  ta  value  *  9] 

USER  FRIENDLINESS  (node  112121)  means  ham  easy  it  is  far  a  user 
ta  operate  the  system  and  get  it  ta  da  mhat  s/he  mants. 

-Host i le  to  user  means  that  the  user  has  ta  adjust  ta  the 
deficiencies  af  the  automated  system.  The  burden  is  an  the 
user  ta  understand  exactly  mhat  ta  da  and  ham  ta  da  it. 
riistakes  are  not  talerated  kindly. 

-Po I i te  to  user  means  the  system  tolerates  same  common  user 
errors,  offers  simple  menus  ar  other  interactive  input  aids, 
includes  a  ‘help*  function  and  other  built  in  training  aids. 

-Gracious  and  accommodating  to  user  needs  means  that  in 
addition  ta  *palite*  the  user  friendly  interface  takes  an  all 
the  burden  af  figuring  aut  horn  the  system  accomplishes  a 
task.  Ula  a  range  af  optional  techniques  (Interactive 
screens,  multiple  mindams,  natural  language,  etc.)  the  user 
only  indicates  mhat  s/he  mants.  fld  hoc  questions  may  be 
osked,  nem  output  displays  can  be  created  dynamically,  etc. 

FUSION  SOPHISTICATION  (node  112123)  means  the  complexity  of  the 
algorithms  and  decision  aids  (saftmare  complexity)  of  the  RDP  system. 
The  lamest  reasonable  level  af  fusion  sophistication  mould  simply  be 
the  ability  ta  put  information  from  multiple,  diverse  sources  into  a 
camman  format  far  subsequent  manual  fusion.  R  CCTU  system,  far 
example,  mould  provide  this  level  af  sophistication.  Though  this  is 
a  useful  and  nan-trivial  level  af  fusion  capability,  me  expect  that 
an  ROP  system  mould  also  provide  same  minimal  apability  ta  combine 
the  information  fram  multiple  sources.  Hence,  ‘simple  algorithms*  is 
the  lamest  defined  level  far  RDP  fusion  sophistication. 

-simple  algorithms  for  fern  data  items  implies  a  variety  af 
morning  and  situation  status  indicators. 

-complex  algorithms  are  comparable  ta  the  sophisticated 
calculations  used  in  processing  missile  morning  data  in  the 
HUC.  Hany  data  items  fram  several  sources  are  combined  ta 
provide  ane  nem  information  item  far  the  decision  maker. 

-decision  aids  mould  guide  the  decision  maker  tamard  nem 
quest  i  oni-hfi-Jliobt  ask  in  a  q^iven  assessed  situation.  Rids 
could  include  automated  checklists  and  a  variety  of  *mhat  if* 
calculations.  Comparisons  betmeen  current  event  information 
and  historic  trends  cauld  be  another  form  af  decision  aid. 
Graphic  display  might  be  an  important  adjunct  ta  this 
capabi I i ty. 
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-expert  systems  aould  outoaot icol ly  decide  ahich  nea 
quest ions  and  euoluations  aere  necessory  to  further  assess 
so»e  ospect  of  the  situotion  ond  tould  gother  supporting  doto 
directly,  without  odditionol  user  guidonce.  Expert  systems 
could  include  pot  tern  recognition  olgorithas,  options  for 
exploining  to  the  user  hoe  porticulor 

euol uot  lono/reconendot  ions  aere  orriued  ot,  etc.  Expert 
progroas  could  run  continuously  on  ony  of  the  coBiond 
center's  OOP  systeas,  ond  contlnuolly  eonitor  inforaotion  ond 
Borning  indicotors  froe  eultiple  sources.  Hoaeuer, 
deuelopeent  of  o  sophisticated  expert  systea,  expert  in 
strotegic  coeBond  ond  control,  is  not  o  trluiol  undertaking 
ond  they  ore  not  likely  to  be  reodily  ouoiloble  in  quontity 
in  the  very  neor  future. 

CRPRCITY  (node  1 12122)  Uhereos  sophistication  is  o  aeosure  of 
quality;  copocity  is  o  aeosure  of  quontity  and  describes  hoa  auch 
fusion  the  fiOP  systea  con  support. 

-Hone  aeons  no  HOP  fusion  copobility.  If  the  leuel  of 
copocity  ■  "Hone"  then  the  leuel  of  Fusion  Doto  Processing  • 
“Ho  flutoaoted  Fusion*.  Hence,  it  is  not  necessory  to  include 
this  leuel  in  the  quest ionno ires. 

-hi ni bo  I  copocity  aeons  thot  the  systea  con  support  o  fixed 
nuaber  of  olgorithas  ahich  continuously  prouide  indicot  ions 
ond  aorning  during  peocetiae  operotions.  There  is  no  support 
for  extreaely  coBplex  olgorithas,  for  expert  systeas,  or  far 
the  increased  inforaotion  load  ossocioted  aith  on  ottock 
scenorio. 

-Li aited  copocity  aeons  thot  not  oil  the  inforaotion  con  be 
coabined  in  oil  the  aoys  thot  seea  reosonoble  for  decision 
aokers  to  look  ot.  Liaited  aoy  aeon  thot  only  one  expert 
systea  is  ouoilable  ond  thot  it  can  deol  aith  only  o  fea 
aspects  of  the  situation.  Uith  o  liaited  systea  users  aust 
be  select iue  about  ahich  questions  they  ask  or  ahich 
indicotors  they  aont  to  aonitor.  "Liaited*  aoy  be  the  result 

of  either  hordaore  liaitotions  or  softaore  liaitotions. 
Hardaare  aay  liait  the  nuaber  ar  complexity  af  caaputatians 
thot  the  systea  con  per fora  in  o  giuen  tiae.  Softaore,  ahich 
is  likely  to  be  the  octuol  constroint  on  the  operotionol 
systeas  of  Interest,  aoy  be  Malted  by  the  difficulty  of 
defining  coaplex  soph i st i coted  olgorithas  ond  decision  oids, 
or  the  tiae  ond  cost  of  deueloping  expert  systeas  for  coaaond 
centers . 

R  aore  rigid  definition  for  “Liaited-  —  Rbility  to  support 
on  o  continuous  doy-to-doy  bosis:  20  siaple  indicotors 

(olgorithas)  that  eoch  coabine  inforaotion  froa  3  different 
sources;  and  10  coaplex  olgorithas  (if  the  RDP  systea  is 
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sophist looted  enough  to  hove  them)  that  combine  multiple 
information  items  from  3  different  sources;  and  1  decision 
olds  (if  the  system  is  sophisticated  enough)  that  help  the 
user  identify  the  mast  important  things  that  should  be  looked 
ot  If  missile  events  occur;  and  only  one  small  expert  system 
thot  takes  3  minutes  to  provide  on  evaluation  of  the  overall 
situot ion. 


-Hot  Liiited  copocity  means  thot  the  above  limitations  ore 
not  present  ond  thot  ony  reasonable  combination  of 
informotion  items  from  many  sources  about  many  events  could 
be  processed  in  o  very  sophisticated  manner  in  near  real 
t  ime. 

DATA  BASE  QUEAY  CAPABILITY  (node  11211  I)  describes  horn  easy  it 
is  to  identify  ond  extract  Information!  from  sources  other  than  the 
MJC,  that  the  command  center  needs  to  fuse  mith  rtUC  Informotion. 
There  ore  tma  steps  to  this:  first,  for  the  command  center  to 
determine  mhat  informotion  it  monts,  mhere  it  is,  ond  horn  to  osk  for 
it  (query  formulation);  ond  second,  for  the  source/ho Ider  of  the 
informotion  to  locate  the  Informotion  ond  extract  o  copy  from  the 
source's  data  base.  It  seems  natural  to  measure  this  capability  by 
the  combined  time  it  takes  to  carry  out  these  tma  steps. 

Clearly,  Ihe  less  time  it  takes  to  get  ony  single  item  of 
informotion,  the  more  complete  the  Informotion  mill  be  for  the  user 
in  the  command  center. 


INTERNAL  COnnUHICATIONS  (node  112112)  is  a  possible  source  of 
delay  in  accessing  data  from  other  sources.  If  the  source  con  only 
be  osked  questions  by  phone,  if  onsmers  must  be  reported  by  voice  or 
hond  delivered,  or  if  slides  must  be  monually  prepored,  then  there 
mill  be  additianol  delays  in  completing  information. 
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APPENDIX  C 


THE  CONCEPT  OF  COMPLETENESS 
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COMPLETENESS  AS  A  MEASURE  FOR  C2  SYSTEMS 

Implementation  of  the  STF  approach  requires  specifying  an  appropriate 
quantitative  measures  at  each  node/questionnaire.  For  several  nodes  we  found 
that  "Completeness  of  Information"  was  a  useful  concept  for  saying  what  a  C2 
system  did.  The  initial  definition  of  Completeness  that  we  used  during  data 
collection  is  attached. 

During  the  course  of  the  experiment  we  found  that  our  naive  definition 
could  capture  only  part  of  the  effect  that  we  were  trying  to  measure.  We 
determined  that  "Completeness"  has  a  content  dependence;  some  information 
was  more  useful  than  other  information.  The  more  useful  information  had 
higher  priority  for  decisionmakers  and  counted  more  toward  completeness. 

For  example,  summary  information  about  a  situation  is  usually  more  valuable 
than  individual  event  messages. 

We  also  found  in  our  discussions  of  information  requirements  for  future 
C2  systems  that  an' explicit  list  of  the  items  of  information  that  a  system 
generated  was  important  for  participants,  both  respondents  and  observers,  to 
understand  clearly  the  concept  of  completeness  and  what  the  questionnaires 
were  about.  Producing  explicit  lists  is  an  additional  cost  of  implementing 
STF. 


GO 


Initiol  definition 


COMPLETENESS  OF  INFORMATION  vs  TIME 

Completeness  Is  concerned  with  how  much  information  about  the  current 
situation  Is  available  to  a  decision  maker  or  other  user.  Information  Is 
complete  if  all  potentially  available  data,  within  the  capacity  of  the 
existing  sensor,  Intel  or  other  collection  systems,  has  been  acquired, 
processed  and  put  into  a  form  that  the  decision  maker  can  use. 

TIME  DEPENDENCE:  At  any  given  time  some  amount  of  raw  data  has 
accumulated  or  could  have  been  accumulated.  If  all  of  that  data  has  been 
processed  into  usable  Information  with  negligible  time  delay,  we  say  that 
the  Information  Is  1 00%  complete.  In  practice  there  will  always  be  some 
minimum  comm  and  processing  delays,  usable  Information  will  always  lag 
the  accumulated  raw  data,  and  information  will  not  be  fully  100%  complete 


TIME  (minutes) 

The  figure  above  shows  the  performance  of  three  possible  information 
processing  systems  as  a  scenario  (series  of  events)  begins  to  develop. 
System  1  Is  a  slow,  reactive  system  that  gradually  collects  available 
Information.  After  a  while  the  system  begins  to  "get  Its  act  together"  and 
catch  up  with  the  data.  After  A  minutes  It  is  presenting  roughly  70%  of  the 
information  In  real  time,  i.e.  as  fast  as  new  data  arrives.  (New  data  is  data 
with  new  Information,  In  contrast  to  repeated  or  redundant  Information.) 

System  2  Is  a  better  system  In  two  respects.  First  It  reacts  more  rapidly  to 
the  new  situation.  Second  it  reaches,  and  maintains,  a  higher  level  of 
completeness  than  System  1. 
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System  3  is  o  prooctive  information  processing  system  with  limited 
capacity.  The  proactive  system  has  already  got  its  act  together  when  the 
new  situation  begins  to  develop,  is  continuously  processing  multiple  data 
sources  and  takes  only  a  minute  to  get  to  a  state  where  it  can  present  80% 
of  the  available  information  in  real  time. 

The  limited  capacity  of  the  system  is  evident  in  the  gradual  decrease  of 
completeness  after  two  minutes.  In  the  scenario  portrayed,  as  more  data 
with  more  information  content  accumulates,  this  information  processing 
system  gets  overloaded  and  begins  to  fall  behind. 

AS  THE  USER  SEES  IT 

The  mission  requirements  analyst,  on  behalf  of  the  user,  is  concerned  with 
specifying  the  overall  system  performance.  Does  the  mission  demand  an 
information  processing  capability  that  gets  its  act  together  in  just  2 
minutes  (System  2)  or  is  it  sufficient  to  have  a  system  that  takes  4  minutes 
to  get  running  (System  1)  and  even  then  provides  no  more  than  70%  complete 
information  in  real  time?  Does  the  mission  require  a  proactive  system? 
Does  the  capacity  of  the  system  have  to  be  big  enough  to  keep  up  with 
arbitrarily  large  amounts  of  raw  data? 

AS  THE  SYSTEM  DEVELOPER  SEES  IT 

The  system  designer/developer  is  concerned  with  what  system  capabilities 
make  an  information  processing  system  better.  How  do  data  processing 
capacity,  data  base  management  systems,  various  display  alternatives,  etc. 
contribute  to  differences  in  system  performance?  How  can  we  change 
System  1  in  the  above  example  into  System  2? 
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APPENDIX  D 


A  TYPICAL  QUESTIONNAIRE 
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Ouestionnoire  1111:  MWC  Information 


Messages 
per  second 
-  1/10 

-  I 

-  10 
-  100 


Messages 
per  second 
-  1/10 
-  1 
-  10 
-  100 


DESCRIPTION 


Time  for 
Modification 

-  Hardcopy  -  3  months 

-  Tabular  -  1  week 

on  CRT  -  1  day 

-  Basic 
Graphics 

-  Enhanced 
Graphics 


This  Questionnaire  addresses  the  relationship  between  the 
■completeness  of  Information  that  Is  produced  by  the  MWC  and  capabilities 
of  systems  within  the  MWC 


Output  Measure 

Completeness  Is  concerned  with  how  much  Information  about  the 
current  situation  Is  available  to  a  decision  maker  or  other  user. 
Information  Is  complete  If  all  potentially  available  data  has  been  acquired, 
processed  and  put  Into  a  form  that  a  decision  maker  can  use. 


Input  Measures 

Data  rate  Is  the  number  of  messages  from  the  sensors  that  arrive  at 
the  MWC  each  second. 

Data  Processing  Capability  Is  measured  by  the  number  of  messages 
from  missile  warning  sensors  (events)  that  can  be  processed  by  the  MWC 
each  second. 

Display  Capability  Is  the  means  available  by  which  missile  warning 
Information  Is  displayed  to  the  MWC  decision  maker  and  staff. 

Hardcoov  means  paper  only,  manually  prepared  hard  copy  (notes, 
reports,  etc.). 

—  Tabular  on  CRT  means  Information  can  be  displayed  on  a  CRT  screen  as 
text  or  simple  tables. 

—  Basic  Graphics  means  Information  can  be  displayed  as  charts, 
diagrams  and  maps  (as  well  as  text). 

—  Enhanced  Graphics  means  real-time.  Interactive  graphics,  with 
continuous  update  of  Information,  ability  to  respond  to  ad  hoc 
requests,  etc. 

System  Currency  means  the  currency  of  the  MWC  system:  the 
frequency  with  which  changes  in  warning  mission  requirements  can  be 
Incorporated  Into  the  system.  Changes  can  Include  changes  In  processing 
algorithms,  display  requirements,  etc. 
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In  the  following  questions  you  are  given  a  single  piece  of  information.  For 
each  question  please  give  your  best  estimate  of  Completeness,  0-100%,  of  MWC 
Information,  bothe  at  ten  minutes  after  alarm  and  at  twenty  minutes  after 
alarm. 

Consider  levels  of  this  factor: 

Missile  Warning  Data  Rate:  the  number  of  valid 
messages  from  the  warning  sensors  that  arrive  at 
the  MWC  each  second. 


LEVEL 


(Short  Form) 


OUTPUT 


@  10  m 

one  tenth  of  a  message  per  second  .1  /  sec 

@  20  m 


one  message  per  second 


1  /  sec 


@  10  m 
@  20  m 


ten  messages  per  second 


10  /  sec 


@  10  m 

@  20  in 


one  hundred  messages  per  second 


100  /  sec 


@  10  ra 
@  20  m 


****************************************************************************** 
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Consider  levels  of  this  factor: 


MWC  Data  Processing  Capability:  the  number  of 
messages  from  the  warning  sensors  (events)  that 
can  be  processed  by  the  MWC  each  second. 


LEVEL 


(Short  Form)  OUTPUT 


@  10  m  _ 

one  tenth  per  second  *1  /  sec 

@  20  m  _ 

@  10  m  _ _ 

one  per  second  1  /  sec 

@  20  m _ _ 

@  10  in  _ 

ten  per  second  10  /  sec 

*  @  20  m  _ _ 

@  10  m  _ 

one  hundred  per  second  100  /  sec 

@  20  m  _ _ 

****************************************************************************** 
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Consider  levels  of  this  factor: 


Missile  Warning  Data  Rate:  the  number  of  valid  messages  from 
the  warning  sensors  that  arrive  at  the  MWC  each  second. 


LEVEL 

(Short  Form) 

OUTPUT 

Paper  Only:  Manually 
prepared  hard  copy 

Paper 

@ 

10 

m 

(notes,  reports,  etc.) 

Only 

@ 

20 

m 

Automated  Tabular  Display: 
Information  can  be  displayed 

automated 

@ 

10 

m 

as  text  or  simple  tables. 

Tabular 

@ 

20 

m 

Basic  Graphics:  Information  can 
be  displayed  as  charts,  diagrams 

Basic 

@ 

10 

m 

and  maps  (as  well  as  text). 

Graphics 

@ 

20 

m 

Enhanced,  Real-time,  Interactive 
Graphics:  continuous  update  of 

Enhanced 

@ 

10 

m 

information,  ad  hoc  interactive 
requests 

Graphics 

@ 

20 

m 

*********************************************************** a  ****************** 


Consider  levels  of  this  factor: 

Currency  of  MWC  System:  the  frequency  with  which  changes  in 
warning  mission  requirements  can  be  incorporated  (processing 
algorithms,  display  requirements,  etc.). 

LEVEL  (Short  Form)  OUTPUT 


@  10  ra 

three  months  three  months 

0  20  m 


0  10  m 

one  week  one  week 

0  20  m 


one  day 


one  day 


0  10  m 
0  20  ra 


****************************************************************************** 
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On  these  pages  you  are  given  two  pieces  of  information  together.  For 
each  question  please  give  your  best  estimate  of  Completeness,  0-100%, 
of  MWC  Information,  both  at  ten  minutes  after  alarm  and  at  twenty 
minutes  after  alarm. 


MISSILE  WARNING  DATA  RATE 


DP  .1  /  sec  1  /  sec  10  /  sec  100  /  sec 

CAPABILITY 

@  10  m  _  _  _  _ _ 

.1  /  sec 

@  20  m 


@  10  m 

1  /  sec 

@  20  ra 


10  /  sec 


@  10  m 
@  20  m 


100  /  sec 


@  10  ra 
(?  20  ra 


****************************************************************************** 
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On  these  pages  you  are  given  two  pieces  of  information  together.  For 
each  question  please  give  your  best  estimate  of  Completeness,  0-100Z, 
of  MWC  Information,  both  at  ten  minutes  after  alarm  and  at  twenty 
minutes  after  alarm. 


MISSILE  WARNING  DATA  RATE 


DISPLAY  .1  /  sec  1  /  sec  10  /  sec  100  /  sec 

CAPABILITY 

0  10  m  _ _ _  _ 

Paper  Only 

0  20  m 


0  10  m 

Automated 

Tabular  0  20  m 


0  10  m 

Basic 

Graphics  0  20  ra 


0  10  m 

Enhanced 

Graphics  0  20  m 


On  these  pages  you  are  given  two  pieces  of  information  together.  For 
each  question  please  give  your  best  estimate  of  Completeness,  0-100%, 
of  MWC  Information,  both  at  ten  minutes  after  alarm  and  at  twenty 
minutes  after  alarm. 


MWC  DATA  PROCESSING  CAPABILITY 


DISPLAY  .1  /  sec  1  /  sec  10  /  sec  100  /  sec 

CAPABILITY 

@  10  m  _  _  _  _ 

Paper  Only 

@  20  m 


@  10  m 

Automated 

Tabular  @  20  m 


@  10  m 

Basic 

Graphics  @  20  m 


@  10  m 

Enhanced 

Graphics  @  20  m 


**££****££******************************************************************** 


70 


On  these  pages  you  are  given  two  pieces  of  information  together.  For 
each  question  please  give  your  best  estimate  of  Completeness,  0-1002, 
of  MWC  Information,  both  at  ten  minutes  after  alarm  and  at  twenty 
minutes  after  alarm. 


CURRENCY  OF  MWC  SYSTEM 


DP 

CAPABILITY 


three 

months 


one 

week 


one 

day 


.1  /  sec 


1  /  sec 


10  /  sec 


100  /  sec 


@  10  m 
@  20  m 

@  10  m 
@  20  m 

@  10  m 
@  20  m 

@  10  m 
@  20  m 


********************************************************** ******************** 


l 
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On  these  pages  you  are  given  two  pieces  of  information  together.  For 
each  question  please  give  your  best  estimate  of  Completeness,  CH100Z, 
of  MWC  Information,  both  at  ten  minutes  after  alarm  and  at  twenty 
minutes  after  alarm. 


CURRENCY  OF  MWC  SYSTEM 


DISPLAY 

three 

one 

one 

CAPABILITY 

months 

week 

day 

@  10  m 

Paper  Only 

@  20  m 


@  10  m 

Automated 

Tabular  @  20  ra 


@  10  m 

Basic 

Graphics  @  20  m 


@  10  ra 

Enhanced 

Graphics  @  20  ra 


**********  *********************************************************  *********** 
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On  these  pages  you  are  given  four  pieces  of  information  together •  For 
each  question  please  give  your  best  estimate  of  Completeness,  0-100%, 
of  MWC  Information,  both  at  ten  nirautes  after  alarm  and  at  twenty 
minutes  after  alarm. 


Message 

Rate 

.1  /  sec 

100  /  sec 

10  /  sec 

10  /  sec 

10  /  sec 

10  /  sec 

1  /  sec 

1  /  sec 

10  /  sec 


DP 

CAPABILITY 

10  /  sec 

10  /  sec 

.1  /  sec 

100  /  sec 

10  /  sec 

10  /  sec 

1  /  sec 

1  /  sec 

10  /  sec 


DISPLAY 

CAPABILITY 

Basic 

Graphics 


Basic 

Graphics 


Basic 

Graphics 


Basic 

Graphics 


Paper 

Only 


Enhanced 

Graphics 


Automated 

Tabular 


Automated 

Tabular 


Basic 

Graphics 


SYSTEM 

CURRENCY 

one 

day 


one 

day 


one 

day 


one 

day 


one 

day 


one 

day 


three 

months 


one 

day 


one 

week 


MWC 

INFO 

@  10  ra  _ 

@  20  ra  _ 

@  10  m  _ 

@  20  ra  _ 

@  10  ra  _ 

@  20  ra  _ 

@  10  ra 
@  20  ra 

@  10  ra  _ 

@  20  ra  _ 

@  10  ra  _ 

@  20  ra  _ 

@  10  m  __ 
@  20  ra  _ 

@  10  ra  _ 

@  20  m  _ 

@  10  ra  _ 

@  20  ra 


On  these  pages  you  are  given  four  pieces  of  information  together. 
For  each  question  please  give  your  best  estimate  of  Completeness, 
0-100Z,  of  MWC  Information,  both  at  ten  minutes  after  alarm  and 
at  twenty  minutes  after  alarm. 


Message  Rate  is  FIXED  at  1  /  sec 


MWC  Display  Capability: 


Automated  Basic 

Tabular  Graphics 


DP 

SYSTEM 

CAPABILITY 

CURRENCY 

three 

@ 

10  m 

1  /  sec 

months 

20  m 

one 

@ 

10  m 

1  /  sec 

day 

<3 

20  m 

****************************************************************************** 

Message  Rate  is  FIXED  at  1  /  sec 


MWC  Display  Capability: 


Automated  Basic 

Tabular  Graphics 

DP  SYSTEM 

CAPABILITY  CURRENCY 


10  /  sec 


three  @  10  ra 

months 

<?  20  ra 


one 

10  /  sec  day 


@  10  m 

<3  20  m 


****************************************** ************************************ 
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Message  Rate  is  FIXED  at  10  /  sec 


MWC  Display  Capability: 


Automated  Basic 

Tabular  Graphics 


DP 

SYSTEM 

CAPABILITY 

CURRENCY 

three 

@ 

10  m 

1  /  sec 

months 

0 

20  m 

one 

0 

10  m 

1  /  sec 

day 

0 

20  m 

****************************************************************************** 


Message  Rate  is  FIXED  at  10  /  sec 


MWC  Display  Capability: 


Automated  Basic 

Tabular  Graphics 


DP 

SYSTEM 

CAPABILITY 

CURRENCY 

three 

0 

10 

in 

10  /  sec 

months 

0 

20 

m 

one 

0 

10 

m 

10  /  sec 

day 

0 

20 

m 

****************************************************************************** 
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APPENDIX  E 


ESTIMATING  PROGRAM  CONTRIBUTION 


i 
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This  appendix  summarizes  our  exploration  of  methods  to  provide  program 
capability  inputs  to  the  bottom  of  the  STF  tree* 

Checklists  are  multiple  choice  responses  that  sharply  restrict  the  range 
of  responses  and  ensure  uniformity  of  data.  Data  so  collected  is  suitable  for 
quantitative  (and  ultimately  automated)  processing.  A  simplified  checklist 
approach  to  Vanguard  data  collection  will  be  useful  even  if  an  STF  scheme  is 
not  attempted.  We  recommend  using  checklists  in  the  Vanguard  data  call. 

ESD  has,  funded  research  to  establish  detailed  checklists  which  enumerate 
many  of  the  system  attributes  that  add  up  to  what  we  have  called  operationally 
oriented  measures  of  capability  (ESD-TR-83-133 ,  Mar  83).  The  checklists  were 
originally  developed  to  aid  in  requirements  definition.  However ,  we  did  not 
discover  any  instances  where  this  approach  is  actually  used. 

Abridged  versions  of  those  checklists  can  be  useful  for  collecting 
relevant  program  information  in  the  Vanguard  data  calls.  Checklists  can  also 
be  developed  for  other  information  of  interest. 

The  following  charts  show  the  checklists  that  were  developed  for  Clarity 
of  Display,  User  Friendliness,  Database  Query  Capability  and  Sophistication. 
The  last  chart  is  the  program  contribution  sheet  which  is  used  to  summarize 
data  about  a  program,  including  information  from  the  checklists,  for  input  to 
VAST. 
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Measuring  Program  Contribution  to  CLARITY  OF  DISPLAY 
(check  ill  that  apply  unless  otherwise  noted) 


Pnysical  Devices 

•  display  area 

(dedicated  to  display  function) 

•  large  display 

(suitable  for  group  viewing: 
greaseboard,  vugraph,  video  projector) 

•  CCTV 

with  two-way  voice 

•  Printed  output 
User.Aids 

•  pre-recorded  material 

(books#briefings,data  files) 

•  pre-formatted  display 

(blank  forms,  background  maps) 

•  training  aids 


•  data  conversion  aids 

Display  Mode 

•  alphanumeric 

free,  formatted,  man-readable 


•  tabular  (check  one) 

(arrayed  display) 


•  basic  graphics 

(simple  maps,  pie/bar  charts) 


•  enhanced  graphics 

(overlays,  split  screens) 


manual 

electric 

elec  Ironic /digital 

_ L 

manual 

_ 2 

electric 

_ 3 

electronic /digital 

electric 

electric 

_ 

U«l 

_ 

♦  graphics 

_ L 

manual 

_ 3 

alec  tronic/digital 

_ L 

manual 

_ 2 

vugraph. 

typewriter 

_ 3 

electronic/digita! 

1 

_ 3 

manual 

electronic  /digital 

_ L 

manual 

_ 3 

automated 

_ L 

formatted 

_ L 

man-re  adable 

fixed 

l 

2 

variable 

2 

3 

manual 

vugraph 

electronic/digital 

2 

3 

vugraph 

electr  on  ic /digital 

•  video  options/color 
(bold, blink, inverse) 


_ L 

color 


_ L 

video  options 
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IntcroctiYcncaa 


•  lypical  time  to  get  ’'standard"  display 

(check  one) 

•  interaction  mode 

•  screen  updating  (check  one) 

•  alarm  mechanisms 


Briefing  Support  (ch«k  on.) 


AC  hoc  capabilities 


_ 2  _ 2 

1  minute  1-15  seconds 

- 2  - 2 

keyboard  menus  mouse/touch 


_ L 

manual 

_ 2 

1  minute 

_ 2 

continuous 

1 

1 

_ L 

visual 

audible  synthetic  voice.etc 

J2  . 

3 

limited 

automated 

enhanced 

automated 

1  ? 

i 

some 

D&MSand 

real-time 

commands  forms  mode  'programming' 


Evolutionary  Flexibility  (check  one) 


_ L  _ 1 

semi-annual  weekly 

modification  upgrade 


TOTAL  PROGRAM  VALUE  FOR  CLARITY  OF  DISPLAY 
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Measuring  Program  Contrioution  to  User  Friend)  iness 


llqpr  Aids 
•  Help  Function 


•  "Standardized"  keyboard/pane) 

•  Special  Function  Keys 

•  "Standardized"  screen  layouts 

Response  to  Input  Errors 

•  Reports  invalid  entries 

•  Allows  edlt/retry 

•  Provides  defaults 

Interaction 

•  Display  Capabilities 

•  with  choice  of  interactive  mode 

•  Natural  Language  modes 

•  Graphics  input 

•  While  computing 

Local  Capabilities 

•  Customized  interface 


some 


Yes 


on-line  on-line 


L 


robust  built  -m 


training 


Yes 


Yes 


user  defined 


L 

visual  alarm 


edit  Inputs 


JL 


basic  entry 


audible  alarm 

reenter/relry 

_ L 

prompt  strings 

2 


Basic  Enhanced  Interactive 

Graphics  Graphics  Graphics 


1 


Yes 


A 


JL 


Basic  mode 


Conversation*!  Mode 


Yes 


JL 


minimizes  delay  keeps  screen  current 


JL 


YtJ 


user  definable 


•  "Desktop"  functions  (clock.calendar .notes,  _ L 

calculator,  etc )  bMic  fUKUon’ 


J. 


♦  local  memory 


TOTAL  PROGRAM  VALUE  FOR  USER  FRIENDLINESS 
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Measuring  Program  Contribution  to  Doto  Bose  Query  Capability 
assuming  intersystem  information  exchange  requirements 
Query  Formulation 

•  Preparation  _ L  _ 2  _ 2 


with  Natural  Language? 

•  User  Aids 

•  Preparation  Time  (check  only  one) 

for  typical  query 


fixed  manual 
request  options 

ad  hoc 
queries 

programmable 

Queries 

_ L 

Yes 

2 

1 

Interactive  Kelp 

Date  Dictionary 

A 

2 

Special  function  Keys 

Interactive  Menu 

JL 

1  minute  or  less 

_ 2 

2-10  seconds 

_ 3 

<  2  seconds 

Information  Retrieval 

•  'Standard'  Intersystem 

Query  Protocol 

•  Compatible  DBMS  among 

connecting  systems 

•  Retrieval  Time  (check  only  one) 

for  typical  record 

•  interoperability  with 

Internal  Communications 


Output  Formatting 

•  Report  Generator 

•  Graphics  Generator 

•  intersystem  Protocol  Standards 


_ 4 

Yes 


_ L 

Yes 


_ L  _ 2  _ 3 

1  minute  or  less  2-10  second)  <  2  seconds 


_ 4  _ 1 

Protocol  Designed  for  Protocol  adapts  to 

Communications  Capacity  system  loading 


_ 1 

Yes 

_ L 

Yes 

_ 1  _ 2  _ L 

for  ASCII  data  files  graphics 


TOTAL  PROGRAM  VALUE  FOR  DATA  BASE  QUERY  CAPABILITY 
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Measuring  Program  Contribution  to  Sophistication 
(software  complexity  of  fusion  processing) 


Algorithm  Complexity 

•  simple  arithmetic  computations 

(  -  spread  sheet  level) 

•  ♦  involved  algebraic  computations 

•  ♦  iterative/recursive  computations 

•  working  memory  requirements: 

-  small 

-  large 

•  logical  complexity: 

-  sequential 

-  conditionaKup  to  5) 

-  multi-conditional(>5) 


•  error  handling  routines  . . 2 

Operator  Interface 

•  -  requires  operator  . .  . . Q 

-  runs  autonomously  . ^  or  ^ . .., . I 

•  runs  continually  (background)  . . L 

•  allows  operator  intervention  . L 

Data  Availability 

•  uses  supplied  data  . . Q 

•  accesses  local  data  base  l 

•  accesses  intersystem  data  L 

•  accesses  large  data  bases  2 

(historical;  plans) 

•  interactive  w-ith  data  base  2 

Decision  Aids 

•  compares  1 

•  shows  trends  2 

•  evaluates  situation  I 

(numerical  computation) 

•  indicates  alternatives  .  . _ I 

•  maintains  checklists  l 

•  responds  to  ad  hoc  questions  . 2 

(What  if?  Compare  these...) 

•  graphics  display  2 


(or) 


{ or ) 

{ or) 
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Expert  Systems 

•  evaluates  situation  L 

•  recognizes  elaborate  patterns 

of  sensor  and  intel  data  2 

•  extrapolates  situation 

(what  next?)  2 

•  generates  alternatives  L 

•  provides  explanations  2 


TOTAL  PROGRAM  VALUE  FOR  SOPHISTICATION 
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SUMMARY  CHECKLIST  FOR  CAPABILITY  INPUTS 


CONTRIBUTION  from  Program 


NODE 

NAME 

1  1  1  1  1 

MWC  Data  Processing 

III  12 

MWC  Clarity  of  Display 

1 1 1 131 

Sensor  Message  Rate 

1  III  33 

Sensor  Voice  Availability 

III  14 

MWC  Flexibility(3mo,wk,day) 

1 1 13 

NORAD/SAC  Data 

1 121 1 1 

Data  Query,  NORAD 

1 121 12 

Internal  Comm,  NORAD 

1 12121 

User  Friendliness,  NORAD 

1 12122 

Capacity  of  NCP  System 

1 12123 

Sophistication  of  NCP  Sys. 

1 1213 

NCP  Clarity  of  Data  Display 

1 12142 

MWC  to  NCP  Data  Comm 

RANGE  CONTRIBUTION 

.1  -100  msg/sec  _ 

qualitative:) -4  _ 

.1  -100  msg/sec  _ 

180-5  seconds  _ 

qualitative:  2-8  _ 

qualitative:  1-4  _ 

180-5  seconds  _ 

180-2  seconds  _ 

qualitative:  1-3  _ 

qualitative  1-4  _ 

qualitative:  1-4  _ 

qualitative  1-4  _ 

qualitative.  1-3  _ 
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